IssueTso.issueTsoCommand incomplete response / data loss

[travatine]

Hi,

I'm using the const result = IssueTso.issueTsoCommand() method and the responses have parts missing. It appears approximately every 1000th line is missing.

My call to IssueTSO:

    const result = await IssueTso.issueTsoCommand(session, "test",  "call 'travist.LOAD(JSONTEST)'");
    if (result.success) {
       const response = result.commandResponse;
       fs.writeFileSync(path.join(context.extensionPath, "out.json"), response);
    }

"JSONTEST" is the following C program, compiled using "ibm xlc"

• i.e.

  xlc test.c ; cp a.out "//'travist.load(JSONTEST)'"

The source code for test.c :

  #include <stdio.h> 

  int main() {        

    printf("{\n");                                                      
    printf(" \"anArray\": \n");                                         
    printf("   [\n");                                                   
    for(int i  = 0 ; i < 10000; i++){                                   
      printf("    {\n");                                                
      printf("       \"element%d\": \"value%d\", \n", i, i);            
      if( i+1 < 10000) {                                                
        printf("    },\n");                                             
      } else {                                                          
        printf("    }\n");                                              
      }                                                                 
    }                                                                   
    printf("   ]\n");                                                   
    printf("}\n");                                                      
  }                                                                     

The resulting output looks like:

    {                                    
     "anArray":                          
       [                                 
        {                                
           "element0": "value0",         
        },                               
        {                                
           "element1": "value1",         
        },                               
        {                                
           "element2": "value2",         
        },                               
        {                                
           "element3": "value3",         
        },                               
        {                                
           "element4": "value4",         
        },
    < -  -   29976 Line(s) not Displayed  > 
        {                                    
            "element9997": "value9997",       
        },                                   
        {                                    
           "element9998": "value9998",       
        },                                   
        {                                    
           "element9999": "value9999",       
        }                                    
      ]                                     
    }                                                                       

However the response from IssueTSO.issueTsoCommand is corrupt / incomplete. at line 1050, 2050, 3050, 4049, 5049, 6049, 7048, 8048 ,... there are missing lines.

In the output the lines are missing:

• the opening "{" of element349,
• the closing "}" of element683
• the line "element1015": "value1015" is missing
• the opening "{" of element1349
• the closing "}" of element1683

• the line "element2016": "value2016" is missing. and more...

  ... { "element347": "value347" }, { "element348": "value348" }, "element349": "value349" }, { "element350": "value350" }, ... { "element681": "value681" }, { "element682": "value682" { "element683": "value683" }, { "element684": "value684" }, ... { "element1349": "value1349" }, "element1350": "value1350" }, { "element1351": "value1351" }, { "element1352": "value1352" }, ... { "element1682": "value1682" }, { "element1683": "value1683" { "element1684": "value1684" }, { "element1685": "value1685" },

[travatine]

I think it is this zosmf api being used: ( https://www.ibm.com/support/knowledgecenter/SSLTBW_2.3.0/com.ibm.zos.v2r3.izua700/izuprog_API_ReceiveMsgTSOAS.htm ) GET /zosmf/tsoApp/tso/

[travatine]

Using the CLI i see the same behaviour: zowe tso start address-space -a test zowe tso send address-space TRAVIST-257-aaeeaaaz --data "call 'travist.TEMP.HCLTOOL.LOAD1(JSONTEST)'"

i.e. the line containing the opening "{" of element7023 is lost:

   {
       "element7020": "value7020",
    },
    {
       "element7021": "value7021",
    },
    {
       "element7022": "value7022",
    },
       "element7023": "value7023",
    },
    {
       "element7024": "value7024",
    },

[travatine]

Hi,

I tried this again today and the corruption in still occurring. e.g. The closing '}' of "element346" is lost:

..., { "element343": "value343", }, { "element344": "value344", }, { "element345": "value345", }, { "element346": "value346", { "element347": "value347", }, { "element348": "value348", }, ...

[travatine]

still occurs today.

[MikeBauerCA]

@travatine, have you tried leveraging the z/OSMF API directly? Just trying to understand if the issue may lie with the CLI or the API and whether we need to open a request with z/OSMF.

[travatine]

Hi @MikeBauerCA ,

I hacked together this script to test zosmf directly: ( It runs the above program using the zosmf tso api, requests all the output, then uses python to pretty print and parse the json)

#!/bin/sh                                                                                                                             
set -eux                                                                                                                              
USER=TRAVIST                                                                                                                          
HOST=pthfae1                                                                                                                          
CURL=/hcl/portedtools/curl7.66/bin/                                                                                                   
PYTHON=/hcl/portedtools/python/2020-05-beta/install/bin/                                                                              
PATH=$PATH:$CURL:$PYTHON                                                                                                              
TSO_CMD="call 'TRAVIST.A4.LOAD(jsontest)'"                                                                                            

read_tso_messages()                                                                                                                   
{                                                                                                                                     
 curl --max-time 2  -k -b cookies.txt \                                                                                               
      -H "Origin: https://${HOST}" \                                                                                                  
  -X 'GET' \                                                                                                                          
  -H 'Connection: keep-alive' \                                                                                                       
  "https://${HOST}:443/zosmf/tsoApp/tso/${servletKey}" > response$1.json                                                              
  iconv -f ibm1047 -t utf8   response$1.json >  response-$1-u.json                                                                    
  #rm  response$1.json                                                                                                                
  if [  -s "response-$1-u.json" ]                                                                                                     
  then                                                                                                                                
    python3 parse-it-u.py  response-$1-u.json  >> response-extracted.json                                                             
  fi                                                                                                                                  
  rm response-$1-u.json                                                                                                               
}                                                                                                                                     

curl -k --user ${USER} https://${HOST}:443/zosmf/services/authenticate  \                                                             
    -H 'Content-Type: application/x-www-form-urlencoded' \                                                                            
      -H "Origin: https://${HOST}" \                                                                                                  
      --cookie-jar cookies.txt                                                                                                        

curl  -k -b cookies.txt \                                                                                                             
      -H "Origin: https://${HOST}" \                                                                                                  
  "https://${HOST}:443/zosmf/tsoApp/tso?proc=IZUFPROC&chset=697&cpage=1047&rows=204&cols=160&rsize=50000&acct=${USER}&ugrp=" \        
  -X 'POST' \                                                                                                                         
  -H 'Connection: keep-alive' \                                                                                                       
  -H 'Content-Length: 0' \                                                                                                            
  -H 'Content-Type: application/x-www-form-urlencoded' \                                                                              
  -H 'Accept: */*' \                                                                                                                  
  -H 'Sec-Fetch-Site: same-origin' \                                                                                                  
  -H 'Sec-Fetch-Mode: cors' \                                                                                                         
  -H 'Sec-Fetch-Dest: empty' > initial.txt                                                                                            

servletKey=$(cat initial.txt | python3 -c 'import json, sys; print(json.loads(sys.stdin.read())["servletKey"]);');                    

echo "servletKey=${servletKey}";                                                                                                      

rm -f  response-extracted.json                                                                                                        

curl  -k -b cookies.txt \                                                                                                             
      -H "Origin: https://${HOST}" \                                                                                                  
  -X 'PUT' \                                                                                                                          
  -H 'Connection: keep-alive' \                  
  "https://${HOST}:443/zosmf/tsoApp/tso/${servletKey}?readReply=true" \                     
  -d "{\"TSO RESPONSE\":{\"VERSION\":\"0100\",\"DATA\":\"${TSO_CMD}\"}}"   > response0.json 
  iconv -f ibm1047 -t utf8   response0.json >  response-0-u.json                            
  rm  response0.json                                                                        
  python3 parse-it-u.py  response-0-u.json  >> response-extracted.json                      
  rm response-0-u.json                                                                      

# read more messages:                                                                       
read_tso_messages 1                                                                         
read_tso_messages 2                                                                         
read_tso_messages 3                                                                         
read_tso_messages 4                                                                         
read_tso_messages 5                                                                         
read_tso_messages 6                                                                         
read_tso_messages 7                                                                         
read_tso_messages 8                                                                         
read_tso_messages 9                                                                         
read_tso_messages 10                                                                        
read_tso_messages 11                                                                        
read_tso_messages 12                                                                        
read_tso_messages 13                                                                        
read_tso_messages 14                                                                        
read_tso_messages 15                                                                        
read_tso_messages 16                                                                        
read_tso_messages 17                                                                        
read_tso_messages 18                                                                        
read_tso_messages 19                                                                        

python3 -m json.tool response-extracted.json                                                                                                                                     

I extract the "data" from the TSO response json with this python script: parse-it.py

import json                                   
import sys                                    
with open(sys.argv[1]) as json_file:          
    data = json.load(json_file)               
    for element in data['tsoData']:           
        if "TSO MESSAGE" in element:          
            msg = element["TSO MESSAGE"]      
            theData = msg['DATA']             
            if "READY" not in theData:        
               print(theData)                 

The resulting "extracted" data is successfully parsed by "python's json.tool" so I think that means all the data was returned correctly from the zosmf api.

The C program I used in the above test was this:

  #include <stdio.h>                                                 

  int main() {                                                       

    printf("{\n");                                                   
    printf(" \"anArray\": \n");                                      
    printf("   [\n");                                                
    for(int i  = 0 ; i < 1000; i++){                                 
      printf("    {\n");                                             
      printf("       \"element%d\": \"value%d\" \n", i, i);          
      if( i+1 < 1000) {                                              
        printf("    },\n");                                          
      } else {                                                       
        printf("    }\n");                                           
      }                                                              
    }                                                                
    printf("   ]\n");                                                
    printf("}\n");                                                   
  }                                                                  

[travatine]

In case it matters I tested on " z/OS 02.04.00 "

[t1m0thyj]

The resulting "extracted" data is successfully parsed by "python's json.tool" so I think that means all the data was returned correctly from the zosmf api.

@awharn and I have tested this with a similar Bash script and get the same error with cURL commands as with Zowe CLI. The data loss seems to be inconsistent - sometimes the response is complete and other times there are one or more gaps in the data.

I noticed that your first C example used to test Zowe CLI prints 30k lines, and the second C example used to test cURL commands prints only 3k lines. Perhaps this makes the error less likely to occur when testing with cURL, since the sample size is smaller.

Although it seems that Zowe CLI is not to blame for the data loss, I'd like to get to the bottom of this issue. @travatine Could you try to run the Bash script again several times with the larger data size (30k lines), and see if the error happens using cURL commands within 10 attempts?

[travatine]

Hi @t1m0thyj , I tried changing the above C program to 10000 iterations, and get a similar result to you - every few runs data is lost.

The location of data loss varies too - in one run it was at:

Expecting property name enclosed in double quotes: line 30017 column 5 (char 450646)

and another:

Expecting ',' delimiter: line 2466 column 5 (char 34755)

To simplify things I combined the above scripts into a single batch job: (I had code page issues with 'bash' so used a regular 'sh' script)

//MAKEIT JOB CLASS=A,NOTIFY=&SYSUID.
//*
// JCLLIB ORDER=CBC.SCCNPRC
//*
// SET HLQ=&SYSUID..TEMP
// SET LOADDSN=&HLQ..LOAD
// SET WORKDIR='/tmp'
//*
//ALLOC EXEC PGM=IEFBR14
//LOAD DD DSN=&LOADDSN.,
// SPACE=(TRK,(7,7,1)),UNIT=SYSALLDA,
// DSNTYPE=LIBRARY,DISP=(MOD,KEEP)
//*
//CB EXEC EDCCB
//BIND.SYSLMOD DD DISP=(SHR,PASS),DSN=&LOADDSN.(RUN)
//COMPILE.SYSIN   DD DATA
  #include <stdio.h>

  int main() {

    printf("{\n");
    printf(" \"anArray\": \n");
    printf("   [\n");
    int num = 10000;
    for(int i  = 0 ; i < num; i++){
      printf("    {\n");
      printf("       \"element%d\": \"value%d\" \n", i, i);
      if( i+1 < num) {
        printf("    },\n");
      } else {
        printf("    }\n");
      }
    }
    printf("   ]\n");
    printf("}\n");
  }
/*
//*
//PYTHONSR EXEC PGM=IEBGENER
//SYSIN    DD DUMMY
//SYSPRINT DD DUMMY
//SYSUT1   DD DATA
import json
import sys
with open(sys.argv[1]) as json_file:
    data = json.load(json_file)
    for element in data['tsoData']:
        if "TSO MESSAGE" in element:
            msg = element["TSO MESSAGE"]
            theData = msg['DATA']
            if "READY" not in theData:
               print(theData)
/*
//SYSUT2  DD PATH='/tmp/parse-it-u.py',
//           PATHDISP=(KEEP,DELETE),
//           PATHOPTS=(OCREAT,ORDWR),
//           PATHMODE=(SIRUSR,SIWUSR,SIRGRP,SIROTH),
//           FILEDATA=TEXT
//*
//RUN EXEC PGM=BPXBATCH,REGION=0M,
// PARM='SH rm /tmp/run.sh'
//STDOUT DD SYSOUT=*
//*
//SCRIPT   EXEC PGM=IEBGENER
//SYSIN    DD DUMMY
//SYSPRINT DD DUMMY
//SYSUT1   DD DATA
#!/bin/sh

set -eux
cd /tmp

USER=TRAVIST
HOST=192.168.21.14
CURL=/apc/rocket/envs/21CSW/bin/
PYTHON=/apc/rocket/envs/21CSW/bin/
PATH=$PATH:$CURL:$PYTHON
TSO_CMD="call 'TRAVIST.TEMP.LOAD(RUN)'"

read_tso_messages()
{
  echo "Read $1";
  OP1='--max-time 1  -k -b cookies.txt -X GET' ;
  URL="https://${HOST}:443/zosmf/tsoApp/tso/${servletKey}";
  curl $OP1  $URL > response$1.json
  iconv -f ibm1047 -t utf8   response$1.json >  response-$1-u.json
  rm  response$1.json
  if [  -s "response-$1-u.json" ]
  then
    python3 parse-it-u.py response-$1-u.json >> response-extracted.json
  fi
  rm response-$1-u.json
}

# Authenticate:

URL="https://$USER:password@$HOST:443/zosmf/services/authenticate";
curl -k --cookie-jar cookies.txt $URL
#iconv -futf8 -tibm1047 cookies.txt

# Start a TSO Session:
OP1="-k -b cookies.txt"
URL1="https://${HOST}:443/zosmf/tsoApp/tso?"
URL2="proc=IZUFPROC&chset=697&cpage=1047&rows=204"
URL3="&cols=160&rsize=50000&acct=${USER}&ugrp="
curl $OP1  $URL1$URL2$URL3 -X POST > initial.txt

# Get the session ID:
PY1="import json, sys;";
PY2='print(json.loads(sys.stdin.read())["servletKey"]);'
servletKey=$(cat initial.txt | python3 -c "$PY1$PY2" );
echo "servletKey=${servletKey}";

# remove concatenated JSON from last run:
rm -f  response-extracted.json

# Run a TSO commands:
H1='-k -b cookies.txt -X PUT'
URL1="https://${HOST}:443/zosmf/tsoApp/tso/${servletKey}?readReply=true"
CMD='{"TSO RESPONSE":{"VERSION":"0100","DATA":"'${TSO_CMD}'"}}'
curl $H1  $URL1 -d "$CMD" | tee response0.json
iconv -f ibm1047 -t utf8   response0.json >  response-0-u.json
rm  response0.json
python3 parse-it-u.py  response-0-u.json  >> response-extracted.json
rm response-0-u.json

# read more messages:
read_tso_messages 1
sleep 1
read_tso_messages 2
sleep 1
read_tso_messages 3
sleep 1
read_tso_messages 4
sleep 1
read_tso_messages 5
sleep 1
read_tso_messages 6
read_tso_messages 7
read_tso_messages 8

read_tso_messages 9
read_tso_messages 10
read_tso_messages 11
read_tso_messages 12
read_tso_messages 13
read_tso_messages 14
read_tso_messages 15
read_tso_messages 16
read_tso_messages 17
read_tso_messages 18
read_tso_messages 19
read_tso_messages 20
read_tso_messages 21
read_tso_messages 22
read_tso_messages 23
read_tso_messages 24
read_tso_messages 25
read_tso_messages 26
read_tso_messages 27
read_tso_messages 28
read_tso_messages 29
read_tso_messages 30
read_tso_messages 31
read_tso_messages 32
read_tso_messages 33
read_tso_messages 34
read_tso_messages 35
read_tso_messages 36
read_tso_messages 37
read_tso_messages 38
read_tso_messages 39
read_tso_messages 40

python3 -m json.tool response-extracted.json  > formatted.json

/*
//SYSUT2  DD PATH='/tmp/run.sh',
//           PATHDISP=(KEEP,DELETE),
//           PATHOPTS=(OCREAT,ORDWR),
//           PATHMODE=(SIRUSR,SIWUSR,SIRGRP,SIROTH),
//           FILEDATA=TEXT
//*
//RUN EXEC PGM=BPXBATCH,REGION=0M,
// PARM='SH chmod +x /tmp/run.sh; /tmp/run.sh'
//STDOUT DD SYSOUT=*

[zFernand0]

Hey all, I know it's been a while since we last provided an update on this issue. FWIW, we do have an open ticket with IBM.

[travatine]

Thanks @zFernand0

[zFernand0]

Hey @travatine,

We got a bit more information to share after doing some more testing.

When we reproduce the issue, we always get these messages in the executing address space (USERID(TSU####):JESMSGLG).

16.38.37 TSU00827 *BPXI039I SYSTEM LIMIT MAXPROCSYS HAS REACHED 85% OF ITS CURRENT CAPACITY OF 200
16.38.37 TSU00827 *BPXI039I SYSTEM LIMIT MAXPROCSYS HAS REACHED 90% OF ITS CURRENT CAPACITY OF 200
16.38.41 TSU00827 *BPXI039I SYSTEM LIMIT MAXPROCSYS HAS REACHED 95% OF ITS CURRENT CAPACITY OF 200
16.38.41 TSU00827 *BPXI039I SYSTEM LIMIT MAXPROCSYS HAS REACHED 100% OF ITS CURRENT CAPACITY OF 200

Could you check to see if you get a message similar to those above?

If you do see these, you could increase the MAXPROCSYS dynamically (i.e. without an IPL) from its default of 200. Just use the SETOMVS command, but be careful 😋

⚠️ Specify only the maximum number of processes you expect z/OS UNIX MVS to support ⚠️

---

Quote

Some storage allocations in the kernel address space are based on the MAXPROCSYS value. A larger value means that more pageable storage will be allocated. Most z/OS UNIX MVS processes use an MVS address space, which uses storage. Therefore, avoid specifying a higher value for MAXPROCSYS than the system can support.

If MAXPROCSYS is set too high relative to the maximum number of initiators the Workload Manager (WLM) can provide, response time delays or failures for fork() or spawn() could occur because users have to wait for initiators.

-. L @ IBM

---

Links

• Understanding the limits of the TSO configuration (BPXPRMxx) for MAXPROCSYS:
  • https://www.ibm.com/docs/en/zos/2.4.0?topic=values-maxprocsys
• Dynamically changing the MAXPROCSYS:
  • https://www.ibm.com/docs/en/zos/2.4.0?topic=values-dynamically-changing-certain-bpxprmxx-parameter
• Using the SETOMVS command: (see the Syntax for details)
  • https://www.ibm.com/docs/pt-br/zos/2.4.0?topic=reference-setomvs-command

[zFernand0]

Next steps to resolve this issue:

• [ ] https://github.com/zowe/docs-site/issues/3671
• [ ] https://github.com/zowe/zowe-cli/issues/2144

We may close this issue in the near future in favor of the two enhancements above 😋