PGM JSON Output Issues

[kadler]

Original report by Danny Roessner (Bitbucket: droessner, GitHub: danny-hcs).

---

Found several issues with JSON output. I'm using this sample RPG:

#!rpg
     H AlwNull(*UsrCtl)

       dcl-ds inputDS qualified;
         in1 varchar(5:2);
         in2 varchar(5:2);
       end-ds;

       dcl-pr Main extpgm;
         inCount int(10);
         input likeds(inputDS) dim(20);
         outCount int(10);
         outputA char(10) dim(20);
         last char(20);
       end-pr;

       dcl-pi Main;
         inCount int(10);
         input likeds(inputDS) dim(20);
         outCount int(10);
         outputA char(10) dim(20);
         last char(20);
       end-pi;

       dcl-s i int(10);
       outCount = inCount;
       for i = 1 to inCount;
         outputA(i) = 'test' + %char(i);
       endfor;
       last = '"quoted" text';
       outCount = i - 1;
       return;  

Sample call:

#!javascript
{"pgm":[
    {"name":"DRTEST01",  "lib":"DROESSNER"},
    {"s": {"name":"inCount", "type":"10i0", "value":1, "by":"in"}},
    {"ds": [{"name":"inputDS","dim":20, "by": "in"},
        {"s":[
            {"name":"in1", "type":"5av2", "value":"i1"},
            {"name":"in2", "type":"5av2", "value":"i2"}
        ]}
    ]},
    {"s": {"name":"outCount", "type":"10i0", "value":1, "setlen":"outputA"}},
    {"s": {"name":"outputA", "type":"10a", "dim":20, "dou": "outCount"}},
    {"s": {"name":"last", "type":"20a", "value":"ll"}}
]}

Sample output:

#!javascript

{"script":[{"pgm":["DRTEST01","DROESSNER",{"inputDS":[[]]},{"outCount":1},{"outputA":["test1","","","","","","","","","","","","","","","","","","",""]},{"last":""quoted" text"}]}]}

Noticing the following issues from this example:

1. inputDS specifies "by": "in", but the output still passes it back
2. outputA array should only be the size of outCount
3. Quotes are not escaped in strings so the output JSON is invalid

Let me know if you need me to provide any additional info.

[kadler]

Original comment by Danny Roessner (Bitbucket: droessner, GitHub: danny-hcs).

---

All items reported are fixed. Marking as resolved.

[kadler]

Original comment by Brian Jerome (Bitbucket: bjerome, GitHub: brianmjerome).

---

I retested this in sg5. The inputDS no longer is returned in the output. Unfortunately I cannot mark this as resolved.

[kadler]

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

---

Have you tried this yet? I would Ike to remove this issue.

[kadler]

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

---

{"inputDS":[[]]} ... still getting inputDS returned when "by":"in" is set.

Ok. Changes to remove empty inputDS.

• Yips Super Driver - tests driver - 1.1.2-sg4 - toolkit remove extra “inputDS” when “by”:”in” (Danny R)

#!bash

$ ./test1000_sql400json32 ../json/j0162_pgm_danny01-ds_dou
input(5000000):
{"pgm":[
    {"name":"DANNY01",  "lib":"DB2JSON"},
    {"s": {"name":"inCount", "type":"10i0", "value":3, "by":"in"}},
    {"ds": [{"name":"inputDS","dim":20, "by": "in"},
        {"s":[
            {"name":"in1", "type":"5av2", "value":"i1"},
            {"name":"in2", "type":"5av2", "value":"i2"}
        ]}
    ]},
    {"s": {"name":"outCount", "type":"10i0", "value":1}},
    {"s": {"name":"outputA", "type":"10a", "dim":20, "dou": "outCount"}},
    {"s": {"name":"last", "type":"20a", "value":"ll"}}
]}

output(122):
{"script":[{"pgm":["DANNY01","DB2JSON",
{"outCount":3},
{"outputA":["test1","test2","test3"]},
{"last":"\"quoted\" text"}]}]}

result:
success (0)

[kadler]

Original comment by Brian Jerome (Bitbucket: bjerome, GitHub: brianmjerome).

---

@rangercairns

2) "dob":"on" ...

Yeah I realized it was the lazy way and would rather just use "dou":"label" so I'm glad it is fixed now in the latest.

[kadler]

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

---

{"inputDS":[[]]} ... still getting inputDS returned when "by":"in" is set.

Mmm ... bit more complex to make "go away". I will look into issue.

[kadler]

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

---

"dou:"count" == 0 issue. Thanks.

Ok. I have a new version with this problem fixed for simple array data limitcount.

Yips Super Driver - test driver - 1.1.2-sg3 - toolkit “dou”:”label” fix count equal zero (Danny R)

#!bash

bash-4.3$ ./test1000_sql400json32 ../json/j0166_pgm_danny01-ds_dou
input(5000000):
{"pgm":[
    {"name":"DANNY01",  "lib":"DB2JSON"},
    {"s": {"name":"inCount", "type":"10i0", "value":0, "by":"in"}},
    {"ds": [{"name":"inputDS","dim":20, "by": "in"},
        {"s":[
            {"name":"in1", "type":"5av2", "value":"i1"},
            {"name":"in2", "type":"5av2", "value":"i2"}
        ]}
    ]},
    {"s": {"name":"outCount", "type":"10i0", "value":0}},
    {"s": {"name":"outputA", "type":"10a", "dim":20, "dou": "outCount"}},
    {"s": {"name":"last", "type":"20a", "value":"ll"}}
]}

output(116):
{"script":[{"pgm":["DANNY01","DB2JSON",
{"inputDS":[[]]},
{"outCount":0},
{"outputA":[]},
{"last":"\"quoted\" text"}]}]}

result:
success (0)

BTW -- You do not have to be concerned about the other php async work in this driver. That work has nothing to do with your current activities. (However, if you are a node toolkit user/author, we should replace current 'bad' toolkit for a 'good' async model based on db2sock work SQL400JsonAsync w/callback).

[kadler]

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

---

Looks like the "dou" on the simple data works for me in sg10

Cool. Hopefully continues on more tests (should work).

when the count > 0. (Note "dob":"on" is needed when the count = 0).

Mmmm ... "dou":"label" should work for associated data parm with count == 0. Aka, you should not have to add "dob":"on".

Basically for readers.

1) "dou":"label" - limits data array elements to 'count' value in associated "label" count parm (0 should work)

2) "dob":"on" - lazy man way to stop array output when reaching first array element blank or zero (nothing to see past blank or zero in this array).

I will need to look at this count == 0 issue. Thanks.

[kadler]

Original comment by Brian Jerome (Bitbucket: bjerome, GitHub: brianmjerome).

---

Looks like the "dou" on the simple data works for me in sg10 using the sample rpg/call (no setlen on the counter) when the count > 0. (Note "dob":"on" is needed when the count = 0).

Still getting inputDS returned when "by":"in" is set.

[kadler]

Original comment by Danny Roessner (Bitbucket: droessner, GitHub: danny-hcs).

---

Appreciate the quick fixes on this again. I will load the new build today and do some more testing.

[kadler]

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

---

update for dob zero (not Danny requested)

Yips Super Driver - tests driver - 1.1.1-sg10 - toolkit fix dob "s" array zero terminate output

[kadler]

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

---

... would be nice to be able to have the ability to specify a counter for simple data

Ok, added dou and dob to simple data ("s").

Yips Super Driver - tests driver - 1.1.1-sg9 - toolkit add output dou and dob “s” data array (Danny R)

option 1) "s" dou - do until other data counter field (request)

This will terminate data "s" array output at other field limit.

#!bash

bash-4.3$ ./test1000_sql400json32 ../json/j0162_pgm_danny01-ds_dou 
input(5000000):
{"pgm":[
    {"name":"DANNY01",  "lib":"DB2JSON"},
    {"s": {"name":"inCount", "type":"10i0", "value":3, "by":"in"}},
    {"ds": [{"name":"inputDS","dim":20, "by": "in"},
        {"s":[
            {"name":"in1", "type":"5av2", "value":"i1"},
            {"name":"in2", "type":"5av2", "value":"i2"}
        ]}
    ]},
    {"s": {"name":"outCount", "type":"10i0", "value":1}},
    {"s": {"name":"outputA", "type":"10a", "dim":20, "dou": "outCount"}},
    {"s": {"name":"last", "type":"20a", "value":"ll"}}
]}

output(139):
{"script":[{"pgm":["DANNY01","DB2JSON",{"inputDS":[[]]},
{"outCount":3},
{"outputA":["test1","test2","test3"]},
{"last":"\"quoted\" text"}]}]}

BTW -- Again, "dou": "outCount", you are using "setlen" wrong, so change your test for outCount limit.

option 2) "s" dob - do until blank/zero

This will terminate data "s" array output at first zero/blank array element.

#!bash

bash-4.3$ ./test1000_sql400json32 ../json/j0163_pgm_danny01-ds_dob 
input(5000000):
{"pgm":[
    {"name":"DANNY01",  "lib":"DB2JSON"},
    {"s": {"name":"inCount", "type":"10i0", "value":3, "by":"in"}},
    {"ds": [{"name":"inputDS","dim":20, "by": "in"},
        {"s":[
            {"name":"in1", "type":"5av2", "value":"i1"},
            {"name":"in2", "type":"5av2", "value":"i2"}
        ]}
    ]},
    {"s": {"name":"outCount", "type":"10i0", "value":1}},
    {"s": {"name":"outputA", "type":"10a", "dim":20, "dob": "on"}},
    {"s": {"name":"last", "type":"20a", "value":"ll"}}
]}

output(139):
{"script":[{"pgm":["DANNY01","DB2JSON",
{"inputDS":[[]]},{"outCount":3},
{"outputA":["test1","test2","test3"]},
{"last":"\"quoted\" text"}]}]}

This needs testing in your environment.

[kadler]

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

---

... would be nice to be able to have the ability to specify a counter for simple data since we are dealing with legacy code where data is passed in an array as described.

Delay. I want to think about effects of change to replicate advance "counting" functions of "ds" into "s". Will let you know.

[kadler]

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

---

Quotes are not escaped in strings so the output JSON is invalid

Ok, I added escape to json string output (i think).

Yips Super Driver - test driver - 1.1.1-sg8 - toolkit added escape for json strings returned (Danny R)

#!bash

bash-4.3$ ./test1000_sql400json32 ../json/j0161_pgm_danny01-ds
input(5000000):
{"pgm":[
    {"name":"DANNY01",  "lib":"DB2JSON"},
    {"s": {"name":"inCount", "type":"10i0", "value":3, "by":"in"}},
    {"ds": [{"name":"inputDS","dim":20, "by": "in"},
        {"s":[
            {"name":"in1", "type":"5av2", "value":"i1"},
            {"name":"in2", "type":"5av2", "value":"i2"}
        ]}
    ]},
    {"s": {"name":"outCount", "type":"10i0", "value":1}},
    {"ds": [{"name":"workAround", "dim":20, "dou": "outCount"}, 
        {"s": {"name":"outputA", "type":"10a"}}
    ]},
    {"s": {"name":"last", "type":"20a", "value":"ll"}}
]}

output(184):
{"script":[{"pgm":["DANNY01","DB2JSON",
{"inputDS":[[]]},
{"outCount":3},
{"workAround":[[{"outputA":"test1"}],[{"outputA":"test2"}],[{"outputA":"test3"}]]},
{"last":"\"quoted\" text"}]}]}

result:
success (0)

This needs more testing in your environment.

[kadler]

Original comment by Danny Roessner (Bitbucket: droessner, GitHub: danny-hcs).

---

It would be nice to be able to have the ability to specify a counter for simple data since we are dealing with legacy code where data is passed in an array as described. I think going back and adding the proposed workaround is not an option due to the scope of the work, but if you are against adding this functionality we can also just add a post processing procedure to strip out the additional data before returning to the client.

Having it built in would be preferred, but is not necessary as there are different ways to work around it.

[kadler]

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

---

Again ... You are proposing a new feature for simple data ("s"). The functions dealing with "dou": "outCount" and "setlen":"outputA" are only available with structures ("ds").

Mmmm ... i don't know about this idea. How bad do you want it???

[kadler]

Original comment by Tony Cairns (Bitbucket: rangercairns, GitHub: rangercairns).

---

You are proposing a new feature for simple data ("s"). The functions dealing with "dou": "outCount" are only available with structures ("ds").

"dou": "outCount" ... not available (only "ds")

No "dou": "outCount" feature for simple data data ("s").

"setlen":"outputA" ... not available (only "ds").

No "setlen":"outputA" feature for simple data data ("s").

You are misusing "setlen" (length, not count)

Even if "setlen" supported for "s" (not supported). You are also misusing "setlen":"outputA", as value result of "outCount" is length 20 * 10 = 200. Aka, "setlen" meant for "length" of a "ds" to help with offset math of many MI and system API interfaces, not "count" of a ds.

More, maybe not "s" ...

Simple dimension data "s", dim(20), usually is considered one-array-simple-type. I am not sure we even want to attempt sub setting the output based on "count".

For example, most times simple "s" array data will be inside of a "ds" structure. This would get messy if we allow "chunks" of the "s" elements to disappear inside a "ds".

Mmmm ... i don't know about this idea. In fact, may be a bad idea for simple types.

work around, use "ds" (not "s")

To control "output count" of a simple data "s", simply surround with a "ds". That is, is no difference between "s":dim(20) and a "ds":dim(20) with one data element.

#!bash

bash-4.3$ ./test1000_sql400json32 ../json/j0161_pgm_danny01-ds
input(5000000):
{"pgm":[
    {"name":"DANNY01",  "lib":"DB2JSON"},
    {"s": {"name":"inCount", "type":"10i0", "value":1, "by":"in"}},
    {"ds": [{"name":"inputDS","dim":20, "by": "in"},
        {"s":[
            {"name":"in1", "type":"5av2", "value":"i1"},
            {"name":"in2", "type":"5av2", "value":"i2"}
        ]}
    ]},
    {"s": {"name":"outCount", "type":"10i0", "value":1}},
    {"ds": [ {"name":"workAround", "dim":20, "dou": "outCount"}, 
        {"s": {"name":"outputA", "type":"10a"}}
    ]},
    {"s": {"name":"last", "type":"20a", "value":"ll"}}
]}

output(138):
{"script":[{"pgm":["DANNY01","DB2JSON",
{"inputDS":[[]]},
{"outCount":1},
{"workAround":[[{"outputA":"test1"}]]},
{"last":""quoted" text"}]}]}

result:
success (0)

Note: "setlen":"outputA" is removed from "name":"outCount" because you are not using "length" feature correctly. Aka, "outCount" result is 20 * 10 = 200 (offset length), not a "count" of elements.