Codepage translation and problem processing response messages in COBOL program

[redwolfo]

I coded a COBOL program using as a guide the sample program HWTHXCB1 posted here to post messages from our batch z/OS environment to external servers.

I am having a series of issues receiving and and parsing the response data. I am not using the JSON parser included with the Toolkit. I only need the raw data coming back from the server.

I have a couple of questions:

1. When sending a message from the mainframe to an outside server (Windows, Unix), do I need to request that the outgoing request data be translated to ASCII or do the MVS services do that automatically? (I don's see HWTHXCB1 doing that. I do understand that the response coming back into the mainframe does need to be translated to EBCDIC.

2. Just for testing, I'm posting a restful service request to one of our CICS regions residing in the same LPAR where my batch job is running. Using a CICS debugging tool, I can see the message being processed and a response message sent back. When the message is received by the Toolkit response exit, the data received is not the output from the CICS service, not EBCDIC and not ASCII.

Any assistance I can get with this will be appreciated; specially question #1. Thank you.

[gorelikg]

Re 1. HTTP/HTTPS Enabler does not translate by default, you're correct in your assumption that your application would need to indicate conversion should be done using HWTHSET API, specifically the HWTH_OPT_TRANSLATE_REQBODY and/or HWTH_OPT_TRANSLATE_RESPBODY options

https://www.ibm.com/docs/en/zos/2.5.0?topic=values-options-requests

Re 2. with out exposing anything, is there anything you can share about the data content? and is the HTTP Status in the 200 range?

[redwolfo]

Hi. I have been debugging this issue some more. Still can't get it to work. I added to my program a HWTH_OPT_TRANSLATE_REQBODY I can't get the sample program to work for me because of security issues. We are blocked from doing non-SSL HTTP requests and when I changed the URL to HTTPS then I get a TLS error. My test program is now posting to a server inside our network but my posting fails with a 406 NOT ACCEPTABLE error. I suspect an issue with the request headers, but I can't see what headers are getting sent from the mainframe.

[ian-burnett]

I can't get the sample program to work for me because of security issues. We are blocked from doing non-SSL HTTP requests and when I changed the URL to HTTPS then I get a TLS error.

At a guess, the server you are connecting to may require connections use at least TLS 1.2, but the System SSL implementation that CWET relies upon allows TLS 1.0.

The Slack example here shows how to set a minimum TLS level - use the HWTH_OPT_SSLVERSION key with the HWTH_SSLVERSION_TLSv12 value. The Slack example is written in Rexx, but hopefully this will translate easily enough into COBOL.

https://github.com/IBM/zOS-Client-Web-Enablement-Toolkit/blob/c1c345ba2ec282b45d80c4587e466c767d78aa35/Example-Slack/slack.rexx#L229

There's also possibly the need to set the HWTH-SSL-USE option to true - I can't remember if CWET will auto-detect an https:// URL.

https://www.ibm.com/docs/en/zos/2.5.0?topic=values-options-connections

There may also be a need to specify which ciphers are valid. Again, the Slack example shows how this is done:

https://github.com/IBM/zOS-Client-Web-Enablement-Toolkit/blob/c1c345ba2ec282b45d80c4587e466c767d78aa35/Example-Slack/slack.rexx#L68

https://github.com/IBM/zOS-Client-Web-Enablement-Toolkit/blob/c1c345ba2ec282b45d80c4587e466c767d78aa35/Example-Slack/slack.rexx#L232

If you need to start debugging the TLS connection further, then you may wish to enable SSL trace. The Slack example shows the property that is needed to be set.

https://github.com/IBM/zOS-Client-Web-Enablement-Toolkit/blob/c1c345ba2ec282b45d80c4587e466c767d78aa35/Example-Slack/slack.rexx#L224

See the comments around line 135 for guidance on how to format an SSL trace. I think that will also allow you to see what's on the wire.

https://github.com/IBM/zOS-Client-Web-Enablement-Toolkit/blob/c1c345ba2ec282b45d80c4587e466c767d78aa35/Example-Slack/slack.rexx#L135

@gorelikg - I hadn't appreciated we are lacking a TLS-enabled COBOL sample. I'll try to put one together, but that won't be this week.

[ian-burnett]

My test program is now posting to a server inside our network but my posting fails with a 406 NOT ACCEPTABLE error. I suspect an issue with the request headers, but I can't see what headers are getting sent from the mainframe.

I think you're probably right with that.

When developing my example, I used the Linux nc command to open up a port and echo whatever data was received. The following command will open up port 38100 on any IP address.

nc -l 0.0.0.0 38100

When that port is accessed from a web browser, I get the following output:

GET / HTTP/1.1
Host: cicsperf.hursley.ibm.com:38100
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:103.0) Gecko/20100101 Firefox/103.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cookie: issiGeoIDSelected=na; issiGeoNameSelected=North%20America; 
...

Obviously this relies on you having access to a Linux machine, plus having security authority to open up a port that's not protected by a firewall.

[gorelikg]

@ian-burnett thank you and that would be amazing!

@redwolfo Ian is exactly right, when you enable ssl (HWTH_OPT_USE_SSL = HWTH_SSL_USE) and don't provide a specific SSL version (HWTH_OPT_SSLVERSION), the ssl version defaults to what System SSL has as it's default which for the past few z/OS releases is TLS 1.0

other things to verify (if you don't have this already ...):

1. You need to provide a keystore with a server certificate(HWTH_OPT_SSLKEY), there are no default certificates provided by the HTTP/HTTPS Enabler or System SSL. The example Ian linked you to also does that, search for keystore.

2. Re, the comment about not seeing the headers the server is sending back to you: if you haven't already turn on unredacted verbose trace settings, set that HWTH_OPT_VERBOSE = HWTH_VERBOSE_UNREDACTED that should generate tracing from the http enabler, versus the HWTH_OPT_SSLTRACE was specific to System SSL

3. this may help with understanding all the nuances regarding security connection through the System SSL path: https://makingdeveloperslivesbetter.wordpress.com/2020/06/03/easy-secure-transfer-on-z-os/

[redwolfo]

Hi. Thank you for the suggestions above. I really appreciate the assistance.

I'm still struggling to get the Toolkit to work for us. As a proof of concept, I am now posting from batch a "hello world" message to a CICS region in the same LPAR where my batch job is running. Using debugging tools I can see that my message, including all headers, are received by the CICS test transaction and a response is created. The batch Toolkit response exit is receiving the response message from CICS and I am able to address the response area.

Here is my challenge. In the sample program HWTHBDYX posted here (part of source data HWTHXCB1) field RESP-BODY-PTR has the address of my response data (with the TSO/Xpediter debugging tool I can see the area contents). I do not need any of the code in HWTHBDYX to do the JSON transform. All I need is to pass RESP-BODY-PTR back to the main program from where the HWTHRQST service was invoked. Any thoughts on that?

My other question is, I would rather not have a separate program to handle the response body. Is it possible to set an option to direct the Toolkit to invoke a section or paragraph within the main program instead of a separate exit program (csect)?

Thank you!! Millie

[blairwyman]

All I need is to pass RESP-BODY-PTR back to the main program from where the HWTHRQST service was invoked. Any thoughts on that?

Oh, no. Don't do that. The RESP-BODY-PTR is only valid while the body exit is active. Once HWTHRQST returns, that RESP-BODY-PTR is toast, and the storage is unpredictable/undefined/dead and gone.

The only way to preserve body data (using this non-streaming response body exit), is to copy it "somewhere" while the exit is active. In C, where the body exit may have visibility to the global C variables by name, the body exit can simply allocate some storage, copy the body into it, and then set a global pointer to the allocated storage. I don't know if COBOL has that notion, but I would not be surprised.

Now there is a lot of value in being able to inspect the body in-situ, and to only surface bits and pieces of it. That is working as designed. If you have modest expectations with respect to a maximum body size, one approach I might suggest would be to pre-allocate a suitably sized buffer and then pass its address to the body exit in the USERDATA field. Then the body exit could simply copy (as much of) the body (as fits), leaving the body exit itself is as "thin" as humanly possible.

Now, just FYI the streaming exit does behave very differently. Since you have to provide the storage for the response body, up front, it is up to you to decide what to do with the body storage when you are done. That's a very different model, but is the only option for very large or unbounded response streams.

Hope this helps.

[redwolfo]

Your explanation makes sense. I'm going to explore passing an area address in USERDATA. I'm just doing a proof of concept, but if we do decide to use the Toolkit, I will probably have to use the streaming model to retrieve large amounts of data.
Thank you!!

[aravind-selvakumar]

Web Enablement toolkit has the option to convert the request from EDCDIC to ASCII and response from ASCII to EDCDIC.

you will need to set it during connection setup process, HWTH_OPT_TRANSLATE_REQBODY HWTH_OPT_TRANSLATE_RESPBODY

Reference: https://www.ibm.com/docs/en/zos/2.3.0?topic=enabler-httphttps-options-values