Closed kadler closed 6 years ago
Original comment by Brian Jerome (Bitbucket: bjerome, GitHub: brianmjerome).
No longer getting a segmentation fault, but now getting a memory fault
. I'll create a new issue for that later.
Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).
Limitless seems overkill ... 15MB should be plenty,
Ok, updated my test version of node db2ia on Yips.
#!bash
bash-4.3$ cd /QOpenSys/QIBM/ProdData/OPS/Node6/lib/node_modules/npm/bin/node-gyp-bin/src/db2ia
bash-4.3$ grep DB2_I5_DB2SOCK_OUT_SIZE dbconn.h
#define DB2_I5_DB2SOCK_OUT_SIZE [15000000 (bb)](https://bitbucket.org/litmis/db2sock/commits/15000000) /* @adc (Danny) */
Original comment by Danny Roessner (Bitbucket: droessner, GitHub: danny-hcs).
Limitless seems overkill. I figured the existing RPGLE limit of 16MB would be the way to go. 15MB should be plenty, but don't see a reason not to match. Any chance you can push out a new build of the experimental driver with a higher limit? 15MB if you prefer.
Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).
Are we done Ishmael/Danny (Moby-Dick)? Or are we going to hunt down the 'limitless memory' white whale of toolkit calls???
Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).
... 16,773,104 bytes (16MB) ... 15,000,000 bytes (15MB) ... who cares???
In fact, may be best to limit slightly under to allow for 'crazy big' to slip a bit here and there.
BTW -- Truth in lending (fine print) ... fairness we could argue for additional toolkit design enabled multiple calls to collect paginated parts of a single PGM, SRVPGM call result.Aka, we could boost ILE memory far beyond PASE range, until the RPG program called was standing on tip, tip toes with memory limits (ILE limits).
Hey man ... this is exactly a discussion about 'too much'. You are getting the picture of the work that would cross into the scripting language toolkit driver ... no free lunch with 'crazy', every man needs to hunt the white whale, or not (Moby Dick).
BTW -- I choose not. But then again, not sure I would have joined Capitan Ahab on the white whale hunt to begin with .... 15MB seems ok.
Original comment by Danny Roessner (Bitbucket: droessner, GitHub: danny-hcs).
16,773,104 bytes? https://www.ibm.com/support/knowledgecenter/SSAE4W_9.5.1/com.ibm.etools.iseries.ilerpg.doc/RNF0376.htm
Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).
good topic for public debate ... limits
Question: That is, when does a toolkit call returning MBs of json become 'too much to expect for a toolkit'?
Basically two extremes exist.
1) Never any limit. We should be able to wrap every data base file on the system into a single json representation. Gigabytes, upon gigabytes, upon gigabytes worth of json going across web to your browser. MU-AH HA ... its alive (Frankenstein parody).
2) A limit. Approximately 15MB should be enough for any rational json based call.
Why did you pick 15MB Ranger (me)?
1) The worst case default 32 bit threaded language (language built in web server)
For example, when you compile a language (node, etc.), 32 bit mode 'small model' default. Total memeory for use, you get one 256 mb segment to share bss (program data), stack, and heap (malloc). When language is threaded (aka, async interfaces), each thread could make a toolkit call at the 'same time. Thereby, a language 'web server' could have 17 concurrent thread callers (256MB/15MB = 17 - each 15 MB buffer).
2) The better case 'huge model' 32 bit threaded language (language built in web server)
'Huge' model 32 bit case we could have roughly 9256MB memory (each 256MB, a segment). Or approximately 17 calls (15MB each) 9 (segments). Thereby, a language 'web server' could have 153 concurrent thread callers.
2) Best case 64 bit ...
Default 64 bit heap is 256MB 256 (segments). Or approximately 17 calls (15MB each) 256 (segments). Thereby, a language 'web server' could have 4352 concurrent thread callers.
Note: Math is not exact here, but you get 'pase memory' idea. Always a limit on a 'toolkit call' in any given language environment. Aka, 'crazy expectation' is quantifiable to some extent (in my opinion).
(Reminder, db2sock is not limited per say. This is a toolkit topic).
Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).
DB2PROCJR ... not limitless
Before we start thinking that DB2PROCJR call is the bees knees (great beyond mom's fresh baked cookies). I think we should point out that DB2PROCJR is not limitless either. Thats is it will return a result set, but total size of the result set is 'limited' by the RPG program returning that data.
#!bash
dcl-s sLast151 int(10) inz(0);
dcl-s Occurs151 packed(5:0) inz(5001);
dcl-ds Output151 occurs(5001);
Out151 char(3000);
end-ds;
// result set of row for concat
exec sql
Set Result Sets Array :Output151 For :Occurs151 Rows;
total size: 5000 * 3000 = 15,000,000 (15 MB roughly)
Of course, you can set these limits higher, and higher, and higher ... until PASE runs out of memory ... but hey man.
There is always some kind of 'sanity' limit. So we will never be able to satisfy people that want to ftp an entire data base to the client over a toolkit ... in json. In fact, I am berry much trying not to say 'crazy users', but, well you get the idea. There is always some type of limit to a toolkit interface.
Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).
I was hoping to get away from managing the PLUG size of the input/output the old toolkit used.
There will always be some limit to output size specified by the 'user code'. In this case the user code is the db2ia driver. The db2sock interface below is traditional CLI-style interface found every db2 driver. In this case the node db2ia 'user code' driver is using the API, so it 'picks' the size (plug size is store procedure thinking).
SQLRETURN SQL400Json(SQLHDBC hdbc,
SQLCHAR * injson,
SQLINTEGER inlen,
SQLCHAR * outjson,
SQLINTEGER outlen)
However, seemingly endless design options in node toolkit makes pinning down a design of a node db2ia 'toolkit' interface a bit difficult.
What do you want me to do with my experimental node db2ia driver?
Original comment by Brian Jerome (Bitbucket: bjerome, GitHub: brianmjerome).
I don't want to have to pass in an output limit. That'd be more code I have to maintain on my end. I was hoping to get away from managing the PLUG size of the input/output the old toolkit used.
Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).
db2sock no limit SQL400JsonAsync(hdbc, jin, ilen, jout, olen)
First, for clarity, you are using json toolkit of db2sock. Which, also means that db2ai is simply a transport of toolkit json via SQL400Json(Async). Further there is no limit on SQL400Json(Async) API. Everything in this discussion is about the experimental node db2ai driver.
experimental node db2ia driver ... size limit
This not a production driver (see next topic ). Currently size limit around 5 MB in experimental node driver (below). This is an arbitrary size to test concept of using SQL400Json a heap variable tmpJsonOut (below).
#!c
#include "PaseCliAsync.h" /* @adc */
#define DB2_I5_DB2SOCK_OUT_SIZE [5000000 (bb)](https://bitbucket.org/litmis/db2sock/commits/5000000) /* @adc */
void DbConn::ToolkitCall(const ARGUMENTS& args) {
Isolate* isolate = args.GetIsolate();
HandleScope scope(isolate);
DbConn* obj = ObjectWrap::Unwrap<DbConn>(args.Holder());
DEBUG("ToolkitCall().\n");
CHECK(obj->connAllocated == false, CONN_NOT_READY, "The Connection handler is not initialized.", isolate)
CHECK(args.Length() != 1, INVALID_PARAM_NUM, "The toolkitSync() method accept only one parameter.", isolate)
String::Utf8Value arg0(args[0]);
SQLCHAR* tmpJsonIn = *arg0;
SQLINTEGER tmpJsonInLen = strlen(tmpJsonIn);
SQLCHAR * tmpJsonOut = (SQLCHAR *) malloc(DB2_I5_DB2SOCK_OUT_SIZE);
SQLINTEGER tmpJsonOutLen = DB2_I5_DB2SOCK_OUT_SIZE;
production node db2ia driver ... limitless
Probably a 'limit/size' input parameter will be needed in production driver.
#!bash
toolkit(jsonIn, limit, callback);
dbconn.toolkit(json, [150000000 (bb)](https://bitbucket.org/litmis/db2sock/commits/150000000),
function (result) {
console.log(result);
}
);
confusion on direction node toolkit design (your issue#10)
However, seemingly endless design options in node toolkit makes pinning down a design of a node db2ia a bit difficult. From other db2sock toolkit issue #10 ... In fact, you froze me to inaction. “Nobody move! I dropped me brain!” – Jack Sparrow (Pirates of Caribbean)
Original report by Brian Jerome (Bitbucket: bjerome, GitHub: brianmjerome).
This is an issue with the db2sock toolkit found at the bottom of the SuperDriver page.
I have test PGM that returns a large result set (DS with 15000 records). I've tested this with the db2sock toolkit (toolkitasync.js) and calling the DB2PROCJR (sql) stored procedure. The latter works fine. With the former I am getting a
Segmentation fault (core dumped)
when I try to return around >11000 records.RPG
Input JSON
Changing the value of
inCount
changes the outCount value.