search

kadler / db2sock-test

1 stars 0 forks source link

Defining multiarray DS json #3

Closed kadler closed 6 years ago

kadler commented 7 years ago

Original report by Teemu Halmela (Bitbucket: teemu_, GitHub: Unknown).

I'm trying to test db2sock with our system. But I have a problem defining a json which contains DS arrays.

For example with a program like this.

     H AlwNull(*UsrCtl)

       dcl-ds itemDS qualified;
            field1 char(5);
            field2 char(5);
            field3 char(5);
            field4 char(5);
       end-ds;

       dcl-pr Main extpgm;
         rows zoned(5:0);
         items likeds(itemDS) dim(20);
         last char(10);
       end-pr;

       dcl-pi Main;
         rows zoned(5:0);
         items likeds(itemDS) dim(20);
         last char(10);
       end-pi;

         dcl-s i int(10);
         for i = 1 to rows;
           items(i).field4 = items(i).field1;
           items(i).field3 = items(i).field2;
         endfor;
         last = 'TEST';

       return;

This json doesn't crash and gives almost the right results.

{"pgm":[
    {"name":"TPGM",  "lib":"DB2JSON"},
    {"s": {"name":"rows", "type":"5s0", "value":2}},
    {"ds": [
        {"s":[
            {"name":"field1", "type":"5a", "value":"ff1"},
            {"name":"field2", "type":"5a", "value":"ff2"},
            {"name":"field3", "type":"5a", "value":""},
            {"name":"field4", "type":"5a", "value":""}
        ]},
        {"s":[
            {"name":"field1", "type":"5a", "value":"gg1"},
            {"name":"field2", "type":"5a", "value":"gg2"},
            {"name":"field3", "type":"5a", "value":""},
            {"name":"field4", "type":"5a", "value":""}
        ]}
    ]},
    {"s": {"name":"last", "type":"10a", "value":""}}
]}
output(201): {"script":[{"pgm":["TPGM","DB2JSON",{"rows":2},{"":{"field1":"TEST"},{"field2":{}},{"field3":"ff2"},{"field4":"ff1"},{"field1":"gg1"},{"field2":"gg2"},{"field3":"gg2"},{"field4":"gg1"}},{"last":{}}]}]}

Weird thing is value TEST goes into DS field1.
Should I define the ds name and dim somewhere? Db2socks is still work in progress but is there currently a way to get this working?

I am testing this on v7.3 using the newest db2sock build by hand.

kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Alrighty then ... i just spent a half day explaining parameter marshalling in ILE RPG sample (*). The concepts are correct. I may have made a finger error here occastionally, but i did a fair job of teaching compiler technology (in ten paragraphs or less).

Happy experimentation.

(*) BTW -- ILE c programs use a concept known as natural alignment for data, so unlike RPG 'packed' structures, c structures can have holes in layout geometry. If you do not understand 'natural alignment' please stick to RPG for your examples in ILE.

kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

So one more 'lesson', well, assuming you are admirably irrepressible and will keep going (damn the torpedoes).

Basically we have two kinds of parameters geometries in RPG (any ILE language). Most common is pass by reference. This means a pointer to the data element 's' or a data structure 'ds'. 

       dcl-pr Main extpgm;
         inCount int(10);
         input likeds(inputDS) dim(20);
         outCount int(10);
         output likeds(outDS) dim(20);
         last char(10);
       end-pr;

argv[]
pointer -> inCount int(10);
pointer -> input likeds(inputDS) dim(20);
pointer -> outCount int(10);
pointer -> output likeds(outDS) dim(20);
pointer -> last char(10);

Therefore, in your quest to set the array elements of inputDS, you must be careful not to introduce an additional pointer (shift the parameters).

#!json

ok ... pointer to one ds 'input'

{"ds": [{"name":"input", "dim": 20},
        {"s":[ 
                 {"name":"in1", "type":"5av2", "value":"a1"},
                 {"name":"in2", "type":"5av2", "value":"a2"}
        ]},
]},

ok ... pointer to input (variations of 'ds' and 's' allowed within outer 'ds')
{"ds": [{"name":"input"},
  {"ds": [{"name":"input1", "dim": 10},
        {"s":[ 
                 {"name":"in1", "type":"5av2", "value":"a1"},
                 {"name":"in2", "type":"5av2", "value":"a2"}
        ]},
  ]},
  {"ds": [{"name":"input2", "dim": 10},
        {"s":[ 
                 {"name":"in1", "type":"5av2", "value":"b1"},
                 {"name":"in2", "type":"5av2", "value":"b1"}
        ]},
  ]},
]},

this BAD ...
bad ... pointer to input1
bad ... pointer to input2

  {"ds": [{"name":"input1", "dim": 10},
        {"s":[ 
                 {"name":"in1", "type":"5av2", "value":"a1"},
                 {"name":"in2", "type":"5av2", "value":"a2"}
        ]},
  ]},
  {"ds": [{"name":"input2", "dim": 10},
        {"s":[ 
                 {"name":"in1", "type":"5av2", "value":"b1"},
                 {"name":"in2", "type":"5av2", "value":"b1"}
        ]},
  ]},

The other type of parameter is by value. I have NOT implemented by value in the ILE prcedure yet (yes, i know how). So let's forget 'value' for this moment as it is fairly rare in RPG programming (only real geeks).

       dcl-pr Main extpgm;
         inCount int(10) value;
         input likeds(inputDS) dim(20);
         outCount int(10) value;
         output likeds(outDS) dim(20);
         last char(10);
       end-pr;

argv[]
value -> inCount int(10) value; (in a register, not memory)
pointer -> input likeds(inputDS) dim(20);
value -> outCount int(10) value;  (in a register, not memory)
pointer -> output likeds(outDS) dim(20);
pointer -> last char(10);
kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

If you are using PHP, you would be much better off using the current PHP toolkit with xmlservice. xmlservice understands all the correct rules, and, php toolkit provides an abstraction that is less likely to lead to errrors in logic (although same logic error truncate rules apply).

So, if you are simply among the 'go faster' than xmlservice crowd, you may want to wait until a php toolkit for this new interface is created.

Just a thought ...

kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Oh boy, you also have another logic error. (Man, I really hate being the bad guy).

In your following code snip, you have a structure 'inputDS', which is declared as parameter input likeds(inputDS) dim(20);

#!c

       dcl-ds inputDS qualified;
            in1 varchar(5:2);
            in2 varchar(5:2);
       end-ds;

       dcl-pr Main extpgm;
         inCount int(10);
         input likeds(inputDS) dim(20);
         outCount int(10);
         output likeds(outDS) dim(20);
         last char(10);
       end-pr;

You MUST account for all 20 elements in the input json. In your json you truncated to dim(5). This means RPG will start eating the other parameters to fill out the 20.

This is right (the only right) ...

#!json

{"ds": [{"name":"input", "dim": 20},

This is wrong ...

#!json

{"ds": [{"name":"input", "dim": 5},

This is a logic error. You simply can not 'truncate' dim(20) to dim(5) passed in parameter.

Why? Basically, your compiled RPG program is expecting input likeds(inputDS) dim(20). When your json only passed dim(5), all the values took whatever was in the memory locations following the 5 provided. In case of toolkit, this means that your RPG program started 'eating' past 'input' and took values from the next parameters outCount, output, last. That is, everything shifted in your program because you made an error declaring dimension size of the arrays (made a boo boo).

In theory, you could try this 'truncate' to dim(5) in RPG, but most likely you get a big fat exception when you touched past the first 5 elements. In IBM i (RPG heap management), pass by reference will not 'eat' into next parameters (like json sample here), instead you will simply walk off the end of the allocation (boom exception). However, IBM i platform is unique in this heap aspect, all other platform, you will simply start writing into the next memory location (aka, like our json toolkit).

In json, all we have is 'weird errors' at runtime ... because ... well ... garbage in lead to garbage out. Actually you are lucky you program did not blow up at runtime.

kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Side ... please continue to try new things. The exploration of ideas, accepted or not, often breeds innovation.

However, as said before, we are really trying to focus on the 'practical' to catch the 80% RPG 'use case'. More exotics like 'overlay' and setting individual array elements as parameter input i consider generally less likely in 'real life' (always exceptions of course).

kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

I have the heart of a teacher. I don't mind an open debate. I may even be wrong at times, but not this time i think. So, i trust you now understand why you can't just list individual elements inside a dimensions structure ('ds' wrong thing). Repeat (boring), 's' elements of a 'ds' simply set initial values of all elements in the 'entire' array. To wit, when you added more elements inside your 'ds' dim(20), you actually got a much larger array, aka, many more elements in each record of the 'ds'. Anyway, if you doubt my logic, please feel free to attempt doing this in RPG using only 'ds' structures ... you will end up with ds inside ds or overlay like my json example (trust me).

kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

bottom line ..

You are making an error. Wrong thinking about what a 'ds' means.

real life ...

First design 'parameter marshalling' philosophy ... generally dimensioned 'parameters like s dim(20) or ds dim(999) are used as output parameters. That is to say, most 'natural' human design RPG interfaces are single data values input and dimensioned arrays of 's' and 'ds' as output. In fact, in RPG, ds.elem(i).value=4 becomes so tedious that if you design your 'input' parameter interfaces this way your consumers/users come looking for you with pitch forks, torches and a long rope.

Overstated (for visual), only exception to this convention is languages like C++ that use objects as passed parameters. Wherein, of course, C++ class is simply a fancy name for a data structure with pointer for methods. Essentially 'encapsulation' acted upon by this->method avoids consumer/user revolt (it's a good thing ... but we RPG compiler guys know better).

just 'tween us geeks ..

So, while i will continue our discussion of 'how to set array values inside a ds', this largely will be an academic discussion between compiler theory geeks.

Most important ... To point, you have a fundamental misunderstanding of defining a data structure. Quickly, 's' or data elements of a 'ds' or data structure are only the default values for the whole dim(20) array. To wit, follows same 'compiler' function/convention as 'traditional' RPG, Therefore, you can only set ONE default for each data element in the inz(*BLANKS), inz(0), and, the default values of the first array element are 'propagated' to the entire 'ds' dim(20) elements (RPG compiler 101).

This is correct ...

#!json

{"pgm":[
    {"name":"HAMELA01",  "lib":"DB2JSON"},
    {"s": {"name":"rows", "type":"5s0", "value":20}},
    {"ds": [{"name":"items","dim":20},
        {"s":[
            {"name":"field1", "type":"5a", "value":"ff1"},
            {"name":"field2", "type":"5a", "value":"ff2"},
            {"name":"field3", "type":"5a", "value":""},
            {"name":"field4", "type":"5a", "value":""}
        ]}
    ]},
    {"s": {"name":"last", "type":"10a", "value":""}}
]}

This is incorrect ...

#!json
    {"ds": [{"name":"input", "dim": 5},
        {"s":[ {"name":"in1", "type":"5av2", "value":"a1"}, {"name":"in2", "type":"5av2", "value":"a2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"b1"}, {"name":"in2", "type":"5av2", "value":"b2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"c1"}, {"name":"in2", "type":"5av2", "value":"c2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"d1"}, {"name":"in2", "type":"5av2", "value":"d2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"e1"}, {"name":"in2", "type":"5av2", "value":"e2"}]}
    ]},

fantasy land we go ...

So, basically, you are proposing a json language with no calculation section. To wit, just using 'ds' and 's', how can i program json (no rude intended, just geek delight)??

We can of course use json to set interior element values of an array of 'ds', but requires a bit of creative compiler thinking. So, all we have to 'know' is that RPG is a 'packed' language. That is, all the elements of an array are 'squished' back-2-back, with no gaps. Therefore we can simply surround any given 'ds with another 'ds and disperse the 's' element in any given location.

Wherein this correct defaults setting (common) ...

#!json

    {"ds": [{"name":"items","dim":20},
        {"s":[
            {"name":"field1", "type":"5a", "value":"ff1"},
            {"name":"field2", "type":"5a", "value":"ff2"},
            {"name":"field3", "type":"5a", "value":""},
            {"name":"field4", "type":"5a", "value":""}
        ]}
    ]},

Can become set the 11th element value idea (your question) ...

#!json
  {"ds": [{"name":"hack",},
    {"ds": [{"name":"items","dim":10},
        {"s":[
            {"name":"field1", "type":"5a", "value":"ff1"},
            {"name":"field2", "type":"5a", "value":"ff2"},
            {"name":"field3", "type":"5a", "value":""},
            {"name":"field4", "type":"5a", "value":""}
        ]}
    ]},
    {"s": [
          {"name":"field1", "type":"5a", "value":"bob"},
          {"name":"field2", "type":"5a", "value":"was"},
          {"name":"field3", "type":"5a", "value":"here"},
          {"name":"field4", "type":"5a", "value":"mary"}
    ]},
    {"ds": [{"name":"items","dim":9},
        {"s":[
            {"name":"field1", "type":"5a", "value":"gg1"},
            {"name":"field2", "type":"5a", "value":"gg2"},
            {"name":"field3", "type":"5a", "value":""},
            {"name":"field4", "type":"5a", "value":""}
        ]}
    ]},
  ]},

and another ...

As json syntax and beauty is in the eye of the beholder ... we could alo offer another RPG-like 'ds' idea like 'overlay'.

#!json

    {"ds": [{"name":"items","dim":20},
        {"s":[
            {"name":"field1", "type":"5a", "value":"ff1"},
            {"name":"field2", "type":"5a", "value":"ff2"},
            {"name":"field3", "type":"5a", "value":""},
            {"name":"field4", "type":"5a", "value":""}
        ]}
    ]},
    {"overlay": [{"where":"items","rec":11},
            [{"name":"field1", "type":"5a", "value":"bob"},
             {"name":"field2", "type":"5a", "value":"was"},
             {"name":"field3", "type":"5a", "value":"here"},
             {"name":"field4", "type":"5a", "value":"mary"}]
    ]},

but not what you did ...

Sorry, but this is just wrong thinking ...

       dcl-ds inputDS qualified;
            in1 varchar(5:2);
            in2 varchar(5:2);
       end-ds;

       dcl-pr Main extpgm;
         inCount int(10);
         input likeds(inputDS) dim(20);
         outCount int(10);
         output likeds(outDS) dim(20);
         last char(10);
       end-pr;

       dcl-pi Main;
         :
         input likeds(inputDS) dim(20);

#!json

    {"ds": [{"name":"input", "dim": 5},
        {"s":[ {"name":"in1", "type":"5av2", "value":"a1"}, {"name":"in2", "type":"5av2", "value":"a2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"b1"}, {"name":"in2", "type":"5av2", "value":"b2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"c1"}, {"name":"in2", "type":"5av2", "value":"c2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"d1"}, {"name":"in2", "type":"5av2", "value":"d2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"e1"}, {"name":"in2", "type":"5av2", "value":"e2"}]}
    ]},

BTW -- Logic error. You MUST (repeat MUST), declare dim(20), NOT dim(5). I will 'teach' why in a later post. For now, think RPG, never pass a smaller 'aggregate' dim(5) into a bigger aggregate dim(20), leaves junk in the elements past 5 (actually machine exception likely). In jspon case, dim(20) starts eating the next parameters data to fill out the entire 20 past 5 (BOOM weird result json).

apologies

Again, I have not had time to document the json.

kadler commented 7 years ago

Original comment by Teemu Halmela (Bitbucket: teemu_, GitHub: Unknown).

There seems to still be something weird going on with dim value and I'm not sure if it is intended.

I've modified the test program a little bit.

     H AlwNull(*UsrCtl)

       dcl-ds inputDS qualified;
            in1 varchar(5:2);
            in2 varchar(5:2);
       end-ds;

       dcl-ds outDS qualified;
            out1 varchar(5:2);
            out2 varchar(5:2);
            out3 varchar(10:2);
       end-ds;

       dcl-pr Main extpgm;
         inCount int(10);
         input likeds(inputDS) dim(20);
         outCount int(10);
         output likeds(outDS) dim(20);
         last char(10);
       end-pr;

       dcl-pi Main;
         inCount int(10);
         input likeds(inputDS) dim(20);
         outCount int(10);
         output likeds(outDS) dim(20);
         last char(10);
       end-pi;

         dcl-s i int(10);
         for i = 1 to inCount;
            output(i).out1 = input(i).in1;
            output(i).out2 = input(i).in2;
            output(i).out3 = input(i).in1 + input(i).in2;
         endfor;
         last = 'TEST';
         outCount = i - 1;
       return;

With this json the results are correct but the input array makes a horrible mess. It copies our input to fill the dim.

input(1000):
{"pgm":[
    {"name":"TPGM",  "lib":"DB2JSON"},
    {"s": {"name":"inCount", "type":"10i0", "value":5}},
    {"ds": [{"name":"input", "dim": 5},
        {"s":[ {"name":"in1", "type":"5av2", "value":"a1"}, {"name":"in2", "type":"5av2", "value":"a2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"b1"}, {"name":"in2", "type":"5av2", "value":"b2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"c1"}, {"name":"in2", "type":"5av2", "value":"c2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"d1"}, {"name":"in2", "type":"5av2", "value":"d2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"e1"}, {"name":"in2", "type":"5av2", "value":"e2"}]}
    ]},
    {"s": {"name":"outCount", "type":"10i0", "value":0}},
    {"ds": [{"name":"output", "dim":10},
        {"s":[ {"name":"out1", "type":"5av2", "value":""}, {"name":"out2", "type":"5av2", "value":""}, {"name":"out3", "type":"10av2", "value":""}]},
    ]},
    {"s": {"name":"last", "type":"10a", "value":""}}
]}

output(1189): {"script":[
    {"pgm":["TPGM","DB2JSON",
        {"inCount":5},
        {"input":[
            [{"in1":"a1"},{"in2":"a2"},{"in1":"b1"},{"in2":"b2"},{"in1":"c1"},{"in2":"c2"},{"in1":"d1"},{"in2":"d2"},{"in1":"e1"},{"in2":"e2"}],
            [{"in1":"a1"},{"in2":"a2"},{"in1":"b1"},{"in2":"b2"},{"in1":"c1"},{"in2":"c2"},{"in1":"d1"},{"in2":"d2"},{"in1":"e1"},{"in2":"e2"}],
            [{"in1":"a1"},{"in2":"a2"},{"in1":"b1"},{"in2":"b2"},{"in1":"c1"},{"in2":"c2"},{"in1":"d1"},{"in2":"d2"},{"in1":"e1"},{"in2":"e2"}],
            [{"in1":"a1"},{"in2":"a2"},{"in1":"b1"},{"in2":"b2"},{"in1":"c1"},{"in2":"c2"},{"in1":"d1"},{"in2":"d2"},{"in1":"e1"},{"in2":"e2"}],
            [{"in1":"a1"},{"in2":"a2"},{"in1":"b1"},{"in2":"b2"},{"in1":"c1"},{"in2":"c2"},{"in1":"d1"},{"in2":"d2"},{"in1":"e1"},{"in2":"e2"}]
        ]},
        {"outCount":5},
        {"output":[
            [{"out1":"a1"},{"out2":"a2"},{"out3":"a1a2"}],
            [{"out1":"b1"},{"out2":"b2"},{"out3":"b1b2"}],
            [{"out1":"c1"},{"out2":"c2"},{"out3":"c1c2"}],
            [{"out1":"d1"},{"out2":"d2"},{"out3":"d1d2"}],
            [{"out1":"e1"},{"out2":"e2"},{"out3":"e1e2"}],
            [{"out1":{}},{"out2":{}},{"out3":{}}],
            [{"out1":{}},{"out2":{}},{"out3":{}}],
            [{"out1":{}},{"out2":{}},{"out3":{}}],
            [{"out1":{}},{"out2":{}},{"out3":{}}],
            [{"out1":{}},{"out2":{}},{"out3":{}}]
        ]},
        {"last":"TEST"}
    ]}
]}

But if we leave the dim out of input or set it to 1, to me it looks more correct.

input(990):
{"pgm":[
    {"name":"TPGM",  "lib":"DB2JSON"},
    {"s": {"name":"inCount", "type":"10i0", "value":5}},
    {"ds": [{"name":"input"},
        {"s":[ {"name":"in1", "type":"5av2", "value":"a1"}, {"name":"in2", "type":"5av2", "value":"a2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"b1"}, {"name":"in2", "type":"5av2", "value":"b2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"c1"}, {"name":"in2", "type":"5av2", "value":"c2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"d1"}, {"name":"in2", "type":"5av2", "value":"d2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"e1"}, {"name":"in2", "type":"5av2", "value":"e2"}]}
    ]},
    {"s": {"name":"outCount", "type":"10i0", "value":0}},
    {"ds": [{"name":"output", "dim":10},
        {"s":[ {"name":"out1", "type":"5av2", "value":""}, {"name":"out2", "type":"5av2", "value":""}, {"name":"out3", "type":"10av2", "value":""}]},
    ]},
    {"s": {"name":"last", "type":"10a", "value":""}}
]}

output(659): {"script":[
    {"pgm":["TPGM","DB2JSON",
        {"inCount":5},
        {"input":[
            {"in1":"a1"},{"in2":"a2"},
            {"in1":"b1"},{"in2":"b2"},
            {"in1":"c1"},{"in2":"c2"},
            {"in1":"d1"},{"in2":"d2"},
            {"in1":"e1"},{"in2":"e2"}
        ]},
        {"outCount":5},
        {"output":[
            [{"out1":"a1"},{"out2":"a2"},{"out3":"a1a2"}],
            [{"out1":"b1"},{"out2":"b2"},{"out3":"b1b2"}],
            [{"out1":"c1"},{"out2":"c2"},{"out3":"c1c2"}],
            [{"out1":"d1"},{"out2":"d2"},{"out3":"d1d2"}],
            [{"out1":"e1"},{"out2":"e2"},{"out3":"e1e2"}],
            [{"out1":{}},{"out2":{}},{"out3":{}}],
            [{"out1":{}},{"out2":{}},{"out3":{}}],
            [{"out1":{}},{"out2":{}},{"out3":{}}],
            [{"out1":{}},{"out2":{}},{"out3":{}}],
            [{"out1":{}},{"out2":{}},{"out3":{}}]
        ]},
        {"last":"TEST"}
    ]}
]}

If we leave dim out from both arrays then the output isn't correct which probably is what should happen.

input(980):
{"pgm":[
    {"name":"TPGM",  "lib":"DB2JSON"},
    {"s": {"name":"inCount", "type":"10i0", "value":5}},
    {"ds": [{"name":"input"},
        {"s":[ {"name":"in1", "type":"5av2", "value":"a1"}, {"name":"in2", "type":"5av2", "value":"a2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"b1"}, {"name":"in2", "type":"5av2", "value":"b2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"c1"}, {"name":"in2", "type":"5av2", "value":"c2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"d1"}, {"name":"in2", "type":"5av2", "value":"d2"}]},
        {"s":[ {"name":"in1", "type":"5av2", "value":"e1"}, {"name":"in2", "type":"5av2", "value":"e2"}]}
    ]},
    {"s": {"name":"outCount", "type":"10i0", "value":0}},
    {"ds": [{"name":"output"},
        {"s":[ {"name":"out1", "type":"5av2", "value":""}, {"name":"out2", "type":"5av2", "value":""}, {"name":"out3", "type":"10av2", "value":""}]},
    ]},
    {"s": {"name":"last", "type":"10a", "value":""}}
]}

output(283): {"script":[
    {"pgm":["TPGM","DB2JSON",
        {"inCount":5},
        {"input":[
            {"in1":"a1"},{"in2":"a2"},
            {"in1":"b1"},{"in2":"b2"},
            {"in1":"c1"},{"in2":"c2"},
            {"in1":"d1"},{"in2":"d2"},
            {"in1":"e1"},{"in2":"e2"}
        ]},
        {"outCount":5},
        {"output":[
            {"out1":"a1"},{"out2":"a2"},{"out3":"a1a2"}
        ]},
        {"last":"TEST"}
    ]}
]}
kadler commented 7 years ago

Original comment by Teemu Halmela (Bitbucket: teemu_, GitHub: Unknown).

Thank you for your speedy help. Things are starting to look better.

I'll try make some more example programs that at least fits to our needs. I'll move to our real programs when we get these smaller ones working first. And if the performance is good this could replace XMLSERVICE which struggles with large arrays of data.

End goal here for us is to call RPG programs from the PHP side, so I should be able to help with that.

kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Ok, i added array of array of records for easier json client parsing.

#!json
bash-4.3$ ./test1000_sql400json32 j0160_pgm_hamela01-ds
input(4096):
{"pgm":[
    {"name":"HAMELA01",  "lib":"DB2JSON"},
    {"s": {"name":"rows", "type":"5s0", "value":20}},
    {"ds": [{"name":"items","dim":20},
        {"s":[
            {"name":"field1", "type":"5a", "value":"ff1"},
            {"name":"field2", "type":"5a", "value":"ff2"},
            {"name":"field3", "type":"5a", "value":""},
            {"name":"field4", "type":"5a", "value":""}
        ]}
    ]},
    {"s": {"name":"last", "type":"10a", "value":""}}
]}

output(1447):
{"script":[{"pgm":["HAMELA01","DB2JSON",{"rows":20},{"items":[
[{"field1":"a1"},{"field2":"b1"},{"field3":"c1"},{"field4":"d1"}],
[{"field1":"a2"},{"field2":"b2"},{"field3":"c2"},{"field4":"d2"}],
[{"field1":"a3"},{"field2":"b3"},{"field3":"c3"},{"field4":"d3"}],
[{"field1":"a4"},{"field2":"b4"},{"field3":"c4"},{"field4":"d4"}],
[{"field1":"a5"},{"field2":"b5"},{"field3":"c5"},{"field4":"d5"}],
[{"field1":"a6"},{"field2":"b6"},{"field3":"c6"},{"field4":"d6"}],
[{"field1":"a7"},{"field2":"b7"},{"field3":"c7"},{"field4":"d7"}],
[{"field1":"a8"},{"field2":"b8"},{"field3":"c8"},{"field4":"d8"}],
[{"field1":"a9"},{"field2":"b9"},{"field3":"c9"},{"field4":"d9"}],
[{"field1":"a10"},{"field2":"b10"},{"field3":"c10"},{"field4":"d10"}],
[{"field1":"a11"},{"field2":"b11"},{"field3":"c11"},{"field4":"d11"}],
[{"field1":"a12"},{"field2":"b12"},{"field3":"c12"},{"field4":"d12"}],
[{"field1":"a13"},{"field2":"b13"},{"field3":"c13"},{"field4":"d13"}],
[{"field1":"a14"},{"field2":"b14"},{"field3":"c14"},{"field4":"d14"}],
[{"field1":"a15"},{"field2":"b15"},{"field3":"c15"},{"field4":"d15"}],
[{"field1":"a16"},{"field2":"b16"},{"field3":"c16"},{"field4":"d16"}],
[{"field1":"a17"},{"field2":"b17"},{"field3":"c17"},{"field4":"d17"}],
[{"field1":"a18"},{"field2":"b18"},{"field3":"c18"},{"field4":"d18"}],
[{"field1":"a19"},{"field2":"b19"},{"field3":"c19"},{"field4":"d19"}],
[{"field1":"a20"},{"field2":"b20"},{"field3":"c20"},{"field4":"d20"}]]},
{"last":"TEST"}]}]}

result:
success (0)
kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

FYI -- There is a lot more work to do with arrays. Specifically, like xmlservice, we will have to add programming convention of MAX and COUNT to limit json returned (popular RPG convention).

Something like json below. Where "max" determines maximum array elements to return, and, "many" will be set to the value returned (by RPG). Therein, there will only be up to 10 rows returned in this example ("enddo":"many"), without any json for array elements 11-20. (XMLSERVICE already works this way for old toolkits). 

#!json
{"pgm":[
    {"name":"HAMELA01",  "lib":"DB2JSON"},
    {"s": {"name":"max", "type":"5s0", "value":10}},
    {"s": {"name":"many", "type":"5s0", "value":0}},
    {"ds": [{"name":"itemDS","dim":20, "enddo":"many"},
        {"s":[
            {"name":"field1", "type":"5a", "value":"ff1"},
            {"name":"field2", "type":"5a", "value":"ff2"},
            {"name":"field3", "type":"5a", "value":""},
            {"name":"field4", "type":"5a", "value":""}
        ]}
    ]},
    {"s": {"name":"last", "type":"10a", "value":""}}
]}
kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Ok, I have a fix for your issue.

Yips Super Driver -- compiled with all ILE parts (DB2JSON *savf)

    libdb400-1.0.5-sg4.zip - (2017–09–27 14:43)
        fix for ds dim(20) - hamela01
        added tests_c compiled tests to zip file (dir included)
        version test

        bash-4.3$ ./tests_c/test9999_driver_version32
        run (trace=on)
        version (1.0.5-sg4)
        success (0)

changed your test just a little ..

#!bash

bash-4.3$ ./test1000_sql400json32 j0160_pgm_hamela01-ds
input(4096):
{"pgm":[
    {"name":"HAMELA01",  "lib":"DB2JSON"},
    {"s": {"name":"rows", "type":"5s0", "value":20}},
    {"ds": [{"name":"itemDS","dim":20},
        {"s":[
            {"name":"field1", "type":"5a", "value":"ff1"},
            {"name":"field2", "type":"5a", "value":"ff2"},
            {"name":"field3", "type":"5a", "value":""},
            {"name":"field4", "type":"5a", "value":""}
        ]}
    ]},
    {"s": {"name":"last", "type":"10a", "value":""}}
]}

output(1408):
{"script":[{"pgm":["HAMELA01","DB2JSON",{"rows":20},
{"itemDS":[
{"field1":"a1"},{"field2":"b1"},{"field3":"c1"},{"field4":"d1"},
{"field1":"a2"},{"field2":"b2"},{"field3":"c2"},{"field4":"d2"},
{"field1":"a3"},{"field2":"b3"},{"field3":"c3"},{"field4":"d3"},
{"field1":"a4"},{"field2":"b4"},{"field3":"c4"},{"field4":"d4"},
{"field1":"a5"},{"field2":"b5"},{"field3":"c5"},{"field4":"d5"},
{"field1":"a6"},{"field2":"b6"},{"field3":"c6"},{"field4":"d6"},
{"field1":"a7"},{"field2":"b7"},{"field3":"c7"},{"field4":"d7"},
{"field1":"a8"},{"field2":"b8"},{"field3":"c8"},{"field4":"d8"},
{"field1":"a9"},{"field2":"b9"},{"field3":"c9"},{"field4":"d9"},
{"field1":"a10"},{"field2":"b10"},{"field3":"c10"},{"field4":"d10"},
{"field1":"a11"},{"field2":"b11"},{"field3":"c11"},{"field4":"d11"},
{"field1":"a12"},{"field2":"b12"},{"field3":"c12"},{"field4":"d12"},
{"field1":"a13"},{"field2":"b13"},{"field3":"c13"},{"field4":"d13"},
{"field1":"a14"},{"field2":"b14"},{"field3":"c14"},{"field4":"d14"},
{"field1":"a15"},{"field2":"b15"},{"field3":"c15"},{"field4":"d15"},
{"field1":"a16"},{"field2":"b16"},{"field3":"c16"},{"field4":"d16"},
{"field1":"a17"},{"field2":"b17"},{"field3":"c17"},{"field4":"d17"},
{"field1":"a18"},{"field2":"b18"},{"field3":"c18"},{"field4":"d18"},
{"field1":"a19"},{"field2":"b19"},{"field3":"c19"},{"field4":"d19"},
{"field1":"a20"},{"field2":"b20"},{"field3":"c20"},{"field4":"d20"}]},
{"last":"TEST"}]}]}

result:
success (0)

The above is valid json returned, but i don't like the format. That is, probably should have the "ds" structure return an array of array records to ease the parsing on the client side. Another small thing to add to the list json parser and output controller (toolkit-parser-json).

BTW -- you can replace the json parser with your own version an call toolkit-base on your own. However, again, the interface is not complete, so probably best to wait until i finish my default included project parsers (josn, xml, etc.).

#!bash

bash-4.3$ cat ../tests_ILE-RPG/hamela01.rpgle 
     H AlwNull(*UsrCtl)

       dcl-ds itemDS qualified;
            field1 char(5);
            field2 char(5);
            field3 char(5);
            field4 char(5);
       end-ds;

       dcl-pr Main extpgm;
         rows zoned(5:0);
         items likeds(itemDS) dim(20);
         last char(10);
       end-pr;

       dcl-pi Main;
         rows zoned(5:0);
         items likeds(itemDS) dim(20);
         last char(10);
       end-pi;

         dcl-s i int(10);
         for i = 1 to rows;
           items(i).field1 = 'a' + %char(i);
           items(i).field2 = 'b' + %char(i);
           items(i).field3 = 'c' + %char(i);
           items(i).field4 = 'd' + %char(i);
         endfor;
         last = 'TEST';

       return;
kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

Mmm ... we are not far enough along to document json interface. So, really, all you have is the few samples in the tests_c directory to run with test1000_sql400json32 (test1000_sql400json64).

The json interface will be for toolkit writers mostly (php, node, ruby, python, etc.). However, you will also be able to call directly using json over some other protocol IPC (fastcgi, ILE CGI, socket, whatever).

Are you a toolkit writer??? What language?

kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

So, i looked quickly at your test ... you have the json wrong ... it would look more like this ... (but dim is not working yet).


       dcl-pr Main extpgm;
         rows zoned(5:0);
         items likeds(itemDS) dim(20);
         last char(10);
       end-pr;
#!json

{"pgm":[
    {"name":"HAMELA01",  "lib":"DB2JSON"},
    {"s": {"name":"rows", "type":"5s0", "value":2}},
    {"ds": [{"name":"items","dim":20},
        {"s":[
            {"name":"field1", "type":"5a", "value":"ff1"},
            {"name":"field2", "type":"5a", "value":"ff2"},
            {"name":"field3", "type":"5a", "value":""},
            {"name":"field4", "type":"5a", "value":""}
        ]}
    ]},
    {"s": {"name":"last", "type":"10a", "value":""}}
]}

In this case 'ds' structure named 'items', has a dim(20) qualifier. The json above would set default values for all 20 items (if working correctly).

items(1).field1 = 'ff1  ';
items(1).field2 = 'ff2  ';
items(1).field3 = '     ';
items(1).field4 = '     ';

items(2).field1 = 'ff1  ';
items(2).field2 = 'ff2  ';
items(2).field3 = '     ';
items(2).field4 = '     ';
:
items(20).field1 = 'ff1  ';
items(20).field2 = 'ff2  ';
items(20).field3 = '     ';
items(20).field4 = '     ';

Hypothetically (art of possible), your settings might look something like this ...

Before we go on ... this whole idea is complete wrong for setting individual values in an input array ... but ... i am trying to lead you to the truth (you made an error).

#!json

{"pgm":[
    {"name":"TPGM",  "lib":"DB2JSON"},
    {"s": {"name":"rows", "type":"5s0", "value":2}},
    {"ds": [{"name":"items","dim":20},
        {"s":[
            {"name":"field1", "type":"5a", "value":"ff1"},
            {"name":"field2", "type":"5a", "value":"ff2"},
            {"name":"field3", "type":"5a", "value":""},
            {"name":"field4", "type":"5a", "value":""}
        ]},
        {"s":[
            {"name":"field1", "type":"5a", "value":"gg1"},
            {"name":"field2", "type":"5a", "value":"gg2"},
            {"name":"field3", "type":"5a", "value":""},
            {"name":"field4", "type":"5a", "value":""}
        ]}
    ]},
    {"s": {"name":"last", "type":"10a", "value":""}}
]}

Wherein, array element 1 and 2 where described in your json, but last default values would be 'continue' for remaining ... or ... perhaps last values would return to *BLANKS as default 

items(1).field1 = 'ff1  ';
items(1).field2 = 'ff2  ';
items(1).field3 = '     ';
items(1).field4 = '     ';

items(2).field1 = 'gg1  ';
items(2).field2 = 'gg2  ';
items(2).field3 = '     ';
items(2).field4 = '     ';
:
perhaps carry last value forward (seems wrong)
:
items(20).field1 = 'gg1  ';
items(20).field2 = 'gg2  ';
items(20).field3 = '     ';
items(20).field4 = '     ';
:
perhaps *BLANK (seems more correct)
:
items(20).field1 = '     ';
items(20).field2 = '     ';
items(20).field3 = '     ';
items(20).field4 = '     ';
kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

BTW -- Thanks for cut/paste of both RPG and the JSON, this will be very helpful.

outlook ...

I forgot to add 'forward looking' commentary (db2sock quarterly/hourly stock report). As with any 'forward looking', mine comes with usual 'doesn't mean anything' legal disclaimer.

So, my belief, if we (you and me), try a bunch of standard JSON parameter configuration combinations, eventually we will hit the 80% 'use case' for common RPG programs. I would encourage a focus on the likely for now (your example is likely), and, try the exotic after we get the basics running.

kadler commented 7 years ago

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

status ...

This project is still under construction (overview pages warning). However, cool, thanks for trying out toolkit in the db2 new driver. I will make effort to respond to your finding as quickly as possible (i do sleep however).

current work ..

I am trying to work on another 'stand-alone' area driver (remove old PASE libdb400.a). That is, we do not really have the base db2 driver tested yet.

proposed methodology to work together ...

Save time (mine and yours) ... This looks similar to another fix while ago (fixed is relative hours, not days). I would like to put 'our joint efforts' on same binary level when you test. Can you please download released (pre-compiled), versions on YIPs.