FALLAI-Denis commented 2 years ago

Hi,

Consider the USS file with explicit encoding 1047 and convert=yes:

J070188:/SYS9/DBB/P001/conf: >ls -lT dbbenv.sh                                         
t IBM-1047    T=on  -rwxr-xr-x   1 BPXROOT  OMVSGRP      762 Sep 25  2020 dbbenv.sh

Consider the zosmf profile Zowe CLI V2:

        "TMVS": {
            "type": "zosmf",
            "properties": {
                "host": "tmvs",
                "port": 8444,
                "rejectUnauthorized": false,
                "encoding": 1147
            },
            "secure": [
                "user",
                "password"
            ]
        },

Consider the 3 zowe commands:

>zowe zos-files download uss-file "/SYS9/DBB/P001/conf/dbbenv.sh" -f ./test-noenc.txt --zosmf-p TMVS
>zowe zos-files download uss-file "/SYS9/DBB/P001/conf/dbbenv.sh" --ec 1047 -f ./test-1047.txt --zosmf-p TMVS
>zowe zos-files download uss-file "/SYS9/DBB/P001/conf/dbbenv.sh" --ec 1147 -f ./test-1147.txt --zosmf-p TMVS

The result obtained demonstrates that Zowe CLI (or z/OSMF) does not respect the tagging of the USS file and applies the encoding option declared on the zosmf profile:

This is not compliant and leads to corruption of USS files when working with Zowe CLI (and actually working with Zowe Explorer which relies on Zowe CLI).

FALLAI-Denis commented 2 years ago

PS : same test with a "819" tagged file

J070188:/SYS9/DBB/P001/conf: >ls -lT rse-dbb.env                                       
t ISO8859-1   T=on  -rwxr-xr-x   1 BPXROOT  OMVSGRP     2570 Sep 25  2020 rse-dbb.env

>zowe zos-files download uss-file "/SYS9/DBB/P001/conf/rse-dbb.env" -f ./rse-dbb-noenc.txt --zosmf-p TMVS
>zowe zos-files download uss-file "/SYS9/DBB/P001/conf/rse-dbb.env" --ec 819 -f ./rse-dbb-819.txt --zosmf-p TMVS
>zowe zos-files download uss-file "/SYS9/DBB/P001/conf/rse-dbb.env" --ec 1047 -f ./rse-dbb-1047.txt --zosmf-p TMVS
>zowe zos-files download uss-file "/SYS9/DBB/P001/conf/rse-dbb.env" --ec 1147 -f ./rse-dbb-1147.txt --zosmf-p TMVS

The conversion is done by Zowe CLI not by z/OSMF?

The X-IBM-BPXK-AUTOCVT header is not used by Zowe CLI? https://www.ibm.com/docs/en/zos/2.4.0?topic=interface-zos-unix-file-utilities

Problem in z/OSMF configuration?

FALLAI-Denis commented 2 years ago

740

FALLAI-Denis commented 2 years ago

Hi,

I can reproduce the problem by direct queries to z/OSMF.

On a 1047 tagged file, autcvt=on:

Letting z/OSMF decide the encoding: good result
By forcing the encoding indicated in the zosmf profile (which Zowe CLI seems to do): wrong result
do a PUT fs request for the chtag information, action list:
Letting z/OSMF decide the encoding: still good result

Conclusion: do not force encoding on a 1047 tagged file.

On a 1147 tagged file, autcvt=on:

Letting z/OSMF decide the encoding: wrong result
By forcing the encoding indicated in the zosmf profile (which Zowe CLI seems to do): good result!
do a PUT fs request for the chtag information, action list:
Letting z/OSMF decide the encoding: still wrong result

Conclusion: force encoding on a 1147 tagged file.

On a 819 (iso-8859-1) tagged file, autcvt=on, things are different...

My observation:

By default, on a z/OSMF session, on a GET fs request I obtain a gzip encoded content
I do a PUT fs request for the chtag information, action list, for the same 819 tagged file (but works for any file)
I redo a GET request on same 819 tagged file, and, miraculously, I obtain a text/plain file, and automatic conversion works well!

I cancel current active STC on z/OS:

New z/OSMF request: back to gzip encoded file

Fun!?

We are running with z/OS 2.4. Content-Encoding gzip is new for z/OS 2.4...

Something wrong in our z/OSMF configuration or IZUFPROC procedure?

Bug in z/OSMF and X-IBM-BPXK-AUTOCVT management? The final "47" (1047 vs 1147) is the cause of the problem (z/OSMF tests the ccsid incorrectly)?

FALLAI-Denis commented 2 years ago

PS : I think problem is in z/OSMF...

Test with a smaller 819 tagged file, file size 577, less than 1024 (not 4096):

First GET: wrong result, get a text/plain file
PUT for chtag:
Second GET: good result
Force encoding to 1047: good result too!!!!! strange???

Same with 1147 tagged file:

First GET: wrong result
PUT for chtag:
Second GET: wrong result too...
Force encoding to 1047: wrong result too...
Force encoding to 1147: good result

FALLAI-Denis commented 2 years ago

Confer #1463

awharn commented 1 year ago

Hello Denis,

The zowe zos-files download uss-file command currently supports USS file tagging, and respects the file tags retrieved from z/OSMF if no encoding or binary option is passed into the command, and encoding and binary are not present on the profiles in the user's configuration file. If encoding is provided in either of these locations, conversion is performed on the file to download it as if it were encoded with that codepage. If binary is specified, the raw data is downloaded. Therefore, we believe this command is operating as expected.

FALLAI-Denis commented 1 year ago

@awharn

But encoding in z/osmf profile is mandatory for mvs files, an apply also to uss files...

traeok commented 5 months ago

Hi @FALLAI-Denis,

I performed some tests with the latest version of Zowe CLI and this issue is mostly resolved. In the screenshot below I ran a few download tests on a file that's tagged as ISO8859-1:

#	`encoding` parameter	`encoding` defined on profile	Result
1	None	None	The tag on the USS file is respected, ISO8859-1 output is returned
2	`1047`	None	The `encoding` parameter is respected, IBM-1047 output returned
3	None	`1047`	Encoding set on the profile took precedence; IBM-1047 output returned
4	`ISO8859-1`	`1047`	The `encoding` parameter takes precedence; ISO8859-1 output returned

(only the first 3 tests are shown in this screenshot)

All of these cases work according to Zowe CLI's current expectations. For case 3, I will need to confirm whether the Zowe CLI team intends to have the profile encoding override the tagged file - but the current behavior is to request the encoding that's explicitly defined on the profile, even if the file has a different tag on USS.

Zowe Explorer has also been updated to act similarly - it uses the encoding provided on the profile unless the file is tagged or if the user explicitly changes the encoding using the "Open with Encoding" feature.

Hope this helps!

FALLAI-Denis commented 5 months ago

Hi @traeok,

I do'nt agree... dowloaded USS files, unless binary transfert is required, must always be converted to UTF-8 on workstation. Encoding is for host, not for local.

For USS file, tag information must always have priority, and if file is not tagged, IBM-1047 apply. Encoding on z/OSMF profile must be ignored for USS file.

For download from USS file:

tagged file: conversion from encoding specified on file to UTF-8
untagged file: conversion from IBM-1047 to UTF-8

For upload to USS file:

if file exists and is tagged, conversion from UTF-8 to tagged encoding
if file exists and is not tagged, conversion from UTF-8 to IBM-1047
if file does'nt exists... do'nt know... IBM-1047 ?

Thanks.

traeok commented 5 months ago

@FALLAI-Denis

I agree that when downloading USS files, it should respect the tag before the encoding on the profile. Since the z/OSMF API for USS files supports data conversion and the feature has been available in Zowe CLI through the --encoding parameter for some time, I believe the team intends to keep that capability for those that use it.

However, I feel that the current behavior for the download and upload functions should be updated to match Zowe Explorer, so that:

it respects the tag on a USS file for its encoding (when --encoding is not specified)
if the tag does not exist, it uses the encoding set on the profile
if there is no encoding set on the profile and the file is untagged, it will use the default encoding set on z/OSMF for USS files

Would the above behavior from Zowe Explorer work for your desired use cases in the CLI? Tagged files would be uploaded and downloaded with the same encoding, and untagged files would be downloaded as the default server encoding if it's not specified as a parameter or on the profile (which is defaulted to IBM-1047 on z/OSMF).

Alternatively, a feature such as #1707 would introduce a separate encoding value for uploads and downloads, allowing you to download files as UTF-8 and upload them with a different encoding.

FALLAI-Denis commented 5 months ago

Hi @traeok,

For USS files, the encoding saved on the profile should never be taken into account as it only applies for MVS datasets which don't use tagging. The implicit encoding of an USS file, when file is not tagged, is always IBM-1047, so it is this encoding that should always be taken by default, and not the Zowe profile encoding. However, it can happen that applications (and not USS commands) create files with an encoding other than IBM-1047 without tagging these files, so we must be able to explicitly force the encoding during transfer, but never by default.

Files downloaded or created on the workstation are always considered to be encoded in UTF-8 (except when transferring in binary mode).

In the absence of explicit encoding on the command, the encoding on download and upload must be consistent by being reversed.

I am not sure I understood your proposal, so I am formulating below what seems to me to be relevant to do.

Concerning Zowe Explorer, in the absence of an explicit conversion request from the operator, a file must be uploaded with the same encoding as that used during download:

if the operator does not force the conversion, the rules below must apply,
if the operator forces the conversion on download, but does not force it on upload, the conversion to be applied during upload is the opposite of the conversion applied during download,
if the operator forces the conversion on upload, the conversion used on download is not taken into account.

For an USS file text transfer without forcing the encoding on the Zowe command:

tagged USS file

download: tag encoding --> UTF-8
upload: UTF-8 --> tag encoding

not tagged USS file

download: IBM-1047 --> UTF-8 (do not consider the encoding saved on the profile)
upload: UTF-8 --> IBM-1047 (do not consider the encoding saved on the profile)

For an USS file text transfer with explicit encoding on the Zowe command (option --encoding):

tagged USS file

download: tag encoding --> UTF-8
upload: UTF-8 --> tag encoding

not tagged USS file

download: --encoding option --> UTF-8
upload: UTF-8 --> --encoding option

For an USS file binary transfer (option --binary):

tagged USS file

download: no conversion
upload: no conversion

not tagged USS file

download: no conversion
upload: no conversion

traeok commented 4 months ago

Hi @FALLAI-Denis,

I apologize for the delayed response. If you would like to upload and download USS files with specific encodings while ignoring the encoding specified on your profile, you can use the .zosattributes feature to specify the paths that you want configured to use a specific local and remote encoding. The --attributes option for zos-files download uss-directory allows you to provide an attribute file to control how download and upload requests are made. This option is also available for the zos-files upload dir-to-uss command.

If the --attributes command is not specified, both the upload dir-to-uss and download uss-directory commands will look for this file. The upload command will check the root of the directory that's being uploaded, and the download command will look at the level above the destination folder.

Here's an example .zosattributes file that you can use as a reference for your use case:

# Format: pattern local-encoding remote-encoding
# Upload all files as IBM-1047 and download as UTF-8 
* UTF-8 IBM-1047

Note that if you are trying to convert characters from UTF-8 to IBM-1047 and the characters are not available in the IBM-1047 character set, the mainframe will replace those characters with the substitution symbol. This behavior cannot be avoided as this is standard practice for encoding/decoding across most software and programming languages - any data that cannot be converted to the desired encoding will be lost during the conversion process.

zowe / zowe-cli

Failure to respect USS file tagging #1431

740