This item was de-coupled from the UTF8 warning work (#988)
The issue / current behavior
The base64 sub-option for return_content does not base64 encode anything. Rather, it 'cat's the dd_output to stdout without making a call to iconv and returns that to the caller.
Note: This behavior caused deprecation warnings due to non-utf8 chars being present in stdout.
It may be advantageous for some users to have the ability to base64 encode the contents of some DD and bring it off platform without facing any encoding or auto-conversion issues. Certificates comes to mind as a use case.
The desired behavior
The dd output collected and returned to the user should reflect the base64 encoded contents of the dd.
The code logic
The zos_mvs_raw module:
parses args
constructs an mvscmd command string
allocates temp datasets (for dd_output and dd_input)
runs mvscmd
collects output from the dds and returns it back to the user.
It is for this last step that the base64 sub-option applies. Until that step, the content is generated and stays on platform. If the content is text, it is easily converted to utf-8 and goes through Ansible smoothly. But if the content is not, there is no real value in attempting to convert it to utf-8, that will only corrupt the data. Moreover, since Ansible will treat content returned by a module as utf-8 text, even in binary form, the data will get corrupted. Base64 encoding the content will protect the underlying byte order and will also flow smoothly though Ansible. From there, the Ansible user would be able to store it into a variable and redirect to a file or possibly to another z/OS platform where the content could be decoded and usable.
The approach
Python provides a base64 library which contains methods to encode and decode bytes.
For handling z/OS UNIX files, the python read-binary file methods work beautifully.
if binary:
with open("{0}".format(quote(name)), "rb") as file:
content = base64.b64encode(file.read()).decode('utf-8')
The content is decoded to utf-8 so that it can be passed back to Ansible without any "auto-conversion".
ZOAU provides a similar python API to read data sets in "binary" mode:
with zoau_io.RecordIO("//{0}".format(quoted_name), "r") as records:
content = base64.b64encode(''.join(records.readrecords()))
However, the data set is read record by record and the method returns a list of bytestrings.
I have not been able to round-trip a load object using the above approach the same way I have been able to round-trip a simple C executable using z/OS UNIX.
ketankelkar
commented
1 month ago
This item was de-coupled from the UTF8 warning work (#988)
The issue / current behavior
The
base64sub-option forreturn_contentdoes not base64 encode anything. Rather, it 'cat's the dd_output to stdout without making a call toiconvand returns that to the caller. Note: This behavior caused deprecation warnings due to non-utf8 chars being present in stdout.It may be advantageous for some users to have the ability to base64 encode the contents of some DD and bring it off platform without facing any encoding or auto-conversion issues. Certificates comes to mind as a use case.
The desired behavior
The dd output collected and returned to the user should reflect the base64 encoded contents of the dd.
The code logic
The
zos_mvs_rawmodule:mvscmdcommand stringdd_outputanddd_input)It is for this last step that the
base64sub-option applies. Until that step, the content is generated and stays on platform. If the content is text, it is easily converted to utf-8 and goes through Ansible smoothly. But if the content is not, there is no real value in attempting to convert it to utf-8, that will only corrupt the data. Moreover, since Ansible will treat content returned by a module as utf-8 text, even in binary form, the data will get corrupted. Base64 encoding the content will protect the underlying byte order and will also flow smoothly though Ansible. From there, the Ansible user would be able to store it into a variable and redirect to a file or possibly to another z/OS platform where the content could be decoded and usable.The approach
Python provides a base64 library which contains methods to encode and decode bytes. For handling z/OS UNIX files, the python read-binary file methods work beautifully.
The content is decoded to utf-8 so that it can be passed back to Ansible without any "auto-conversion".
ZOAU provides a similar python API to read data sets in "binary" mode:
However, the data set is read record by record and the method returns a list of bytestrings. I have not been able to round-trip a load object using the above approach the same way I have been able to round-trip a simple C executable using z/OS UNIX.