Properly support UTF-8 files

[nickrayjones]

Setting a file loaded from git with .gitattributes zos-working-tree-encoding=utf-8 results in a file that is tagged ISO8859-1

This cause trouble for programs that read the file tag and assume some conversion is needed.

Inside of git, all files are UTF-8, and so in the file system setting zos-working-tree-encoding=utf-8 should result in no encoding changes.

Files tagged zos-working-tree-encoding=utf-8 should result in a UTF-8 tag and the file not being re-encoded.

[depemco]

We are switching from the Rocket Git to the Z Open Tools Git, where we are having this same issue.

With the Rocket Git, the encoding of UTF-8 is not changed. With the Z Open Tools Git, it changes to ISO8859-1 which results in problems.

How is it we tell in .gitattributes with zos-working-tree-encoding and working-tree-encoding to use UTF-8, but it still tags the files as ISO8859-1?

[IgorTodorovskiIBM]

We are switching from the Rocket Git to the Z Open Tools Git, where we are having this same issue.

With the Rocket Git, the encoding of UTF-8 is not changed. With the Z Open Tools Git, it changes to ISO8859-1 which results in problems.

How is it we tell in .gitattributes with zos-working-tree-encoding and working-tree-encoding to use UTF-8, but it still tags the files as ISO8859-1?

Hey @depemco , are you able to reveal the problems you're facing? Are they with respect to Jenkins?

Here are my findings with Jenkins: ` This simple Java program reads a file as input and prints it out to the screen:

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List;

public class ReadFile {

    public static void main(String[] args) {
        String filePath = "your_file_path.txt";

        try {
            Path path = Paths.get(filePath);
            List<String> lines = Files.readAllLines(path, StandardCharsets.UTF_8);

            // Print each line from the file
            for (String line : lines) {
                System.out.println(line);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Here's my file. initially tagged as 819:

[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ cat your_file_path.txt
asdf
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ ls -lT your_file_path.txt
t ISO8859-1   T=on  -rw-rw-r--   1 ITODORO  CDEV           5 Nov 10 14:26 your_file_path.txt

Here's the result with a file tag as ASCII (819) with autocvt = ON. Clearly, Java is trying to auto-convert from 819 to 1047:

[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=ON
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ java  ReadFile
java.nio.charset.MalformedInputException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:292)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:378)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:211)
        at java.io.InputStreamReader.read(InputStreamReader.java:205)
        at java.io.BufferedReader.fill(BufferedReader.java:172)
        at java.io.BufferedReader.readLine(BufferedReader.java:335)
        at java.io.BufferedReader.readLine(BufferedReader.java:400)
        at java.nio.file.Files.readAllLines(Files.java:3216)
        at ReadFile.main(ReadFile.java:16)

If I turn OFF auto-conversion it works:

[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=OFF
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ java  ReadFile
asdf

Now, let's try UTF-8 tagged files:

[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ chtag -tc UTF-8 your_file_path.txt
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=ON
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ java  ReadFile
asdf

As expected, it works because _BPXK_AUTOCVT=ON, and in this mode, UTF8 tagged files are not auto-converted. If we set _BPXK_AUTOCVT=ALL, will it work? Same issue as above:

[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=ALL
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ java  ReadFile
java.nio.charset.MalformedInputException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:292)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:378)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:211)
        at java.io.InputStreamReader.read(InputStreamReader.java:205)
        at java.io.BufferedReader.fill(BufferedReader.java:172)
        at java.io.BufferedReader.readLine(BufferedReader.java:335)
        at java.io.BufferedReader.readLine(BufferedReader.java:400)
        at java.nio.file.Files.readAllLines(Files.java:3216)
        at ReadFile.main(ReadFile.java:16)
And finally, turning off autoconvert also works:
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=OFF
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ java  ReadFile
asdf

So I think ultimately you want auto-conversion disabled to get reliable behaviour. Either you can set _BPXK_AUTOCVT=OFF, or tag your files as binary.

You can also probably add this before the load call in groovy:

env._BPXK_AUTOCVT = OFF

---

Also, here's some issues I found when tagging files as UTF-8 as opposed to 819:

UTF-8 tagged files seem to work fine with ASCII programs. At least those that I've tried in z/OS Open Tools. But EBCDIC programs like /bin/cat have a hard time understanding them:

[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=ON
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ chtag -tc UTF8 a.txt
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ ls -lT a.txt
t UTF-8       T=on  -rw-rw-r--   1 ITODORO  CDEV          15 Nov 10 13:40 a.txt
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ cat a.txt # using zopen's cat
😓
😰
😻
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ /bin/cat a.txt # using system cat
0�ql�0�q��0�q��[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$

# now let's try with _BPXK_AUTOCVT=ALL
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=ALL
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ /bin/cat a.txt # hmm, that doesn't look good
cat: FSUM6180 file "a.txt": EDC5122I Input/output error.
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ chtag -tc 819 a.txt
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ cat a.txt # looks fine with 819
😓
😰
😻
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ /bin/cat a.txt # also looks fine with 819
😓
😰
😻

[IgorTodorovskiIBM]

One solution is to add an option that can toggle this behaviour as there are some relying on 819 tagged files and other that require UTF-8 tagged files

[depemco]

We are switching from the Rocket Git to the Z Open Tools Git, where we are having this same issue. With the Rocket Git, the encoding of UTF-8 is not changed. With the Z Open Tools Git, it changes to ISO8859-1 which results in problems. How is it we tell in .gitattributes with zos-working-tree-encoding and working-tree-encoding to use UTF-8, but it still tags the files as ISO8859-1?

Hey @depemco , are you able to reveal the problems you're facing? Are they with respect to Jenkins?

Here are my findings with Jenkins: ` This simple Java program reads a file as input and prints it out to the screen:

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.List;

public class ReadFile {

    public static void main(String[] args) {
        String filePath = "your_file_path.txt";

        try {
            Path path = Paths.get(filePath);
            List<String> lines = Files.readAllLines(path, StandardCharsets.UTF_8);

            // Print each line from the file
            for (String line : lines) {
                System.out.println(line);
            }
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
}

Here's my file. initially tagged as 819:

[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ cat your_file_path.txt
asdf
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ ls -lT your_file_path.txt
t ISO8859-1   T=on  -rw-rw-r--   1 ITODORO  CDEV           5 Nov 10 14:26 your_file_path.txt

Here's the result with a file tag as ASCII (819) with autocvt = ON. Clearly, Java is trying to auto-convert from 819 to 1047:

[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=ON
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ java  ReadFile
java.nio.charset.MalformedInputException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:292)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:378)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:211)
        at java.io.InputStreamReader.read(InputStreamReader.java:205)
        at java.io.BufferedReader.fill(BufferedReader.java:172)
        at java.io.BufferedReader.readLine(BufferedReader.java:335)
        at java.io.BufferedReader.readLine(BufferedReader.java:400)
        at java.nio.file.Files.readAllLines(Files.java:3216)
        at ReadFile.main(ReadFile.java:16)

If I turn OFF auto-conversion it works:

[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=OFF
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ java  ReadFile
asdf

Now, let's try UTF-8 tagged files:

[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ chtag -tc UTF-8 your_file_path.txt
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=ON
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ java  ReadFile
asdf

As expected, it works because _BPXK_AUTOCVT=ON, and in this mode, UTF8 tagged files are not auto-converted. If we set _BPXK_AUTOCVT=ALL, will it work? Same issue as above:

[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=ALL
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ java  ReadFile
java.nio.charset.MalformedInputException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:292)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:378)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:211)
        at java.io.InputStreamReader.read(InputStreamReader.java:205)
        at java.io.BufferedReader.fill(BufferedReader.java:172)
        at java.io.BufferedReader.readLine(BufferedReader.java:335)
        at java.io.BufferedReader.readLine(BufferedReader.java:400)
        at java.nio.file.Files.readAllLines(Files.java:3216)
        at ReadFile.main(ReadFile.java:16)
And finally, turning off autoconvert also works:
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=OFF
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ java  ReadFile
asdf

So I think ultimately you want auto-conversion disabled to get reliable behaviour. Either you can set _BPXK_AUTOCVT=OFF, or tag your files as binary.

You can also probably add this before the load call in groovy:

env._BPXK_AUTOCVT = OFF

Also, here's some issues I found when tagging files as UTF-8 as opposed to 819:

UTF-8 tagged files seem to work fine with ASCII programs. At least those that I've tried in z/OS Open Tools. But EBCDIC programs like /bin/cat have a hard time understanding them:

[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=ON
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ chtag -tc UTF8 a.txt
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ ls -lT a.txt
t UTF-8       T=on  -rw-rw-r--   1 ITODORO  CDEV          15 Nov 10 13:40 a.txt
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ cat a.txt # using zopen's cat
😓
😰
😻
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ /bin/cat a.txt # using system cat
0�ql�0�q��0�q��[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$

# now let's try with _BPXK_AUTOCVT=ALL
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ export _BPXK_AUTOCVT=ALL
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ /bin/cat a.txt # hmm, that doesn't look good
cat: FSUM6180 file "a.txt": EDC5122I Input/output error.
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ chtag -tc 819 a.txt
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ cat a.txt # looks fine with 819
😓
😰
😻
[ITODORO@ZOSCAN2B ~/projects/fzfport/meta]$ /bin/cat a.txt # also looks fine with 819
😓
😰
😻

Hi @IgorTodorovskiIBM. I am just the person who installs the software and provides it to the other users of the system. The users tell me there is a difference between the Rocket Git and the Z Open Tools git, as I described earlier.

I myself use the Z Open Tools git without any problems, but I don't use UTF-8 as encoding. But other teams have switched completely to UTF-8 in the past, when the Rocket Git supported it. The reason behind this I think is because of the other systems they work with are also in UTF-8. They don't feel like converting all their scripting again, because it would be a lot of work ánd UTF-8 should just be a supported encoding as it is one of the most popular encodings of the moment.

One solution is to add an option that can toggle this behaviour as there are some relying on 819 tagged files and other that require UTF-8 tagged files

This sounds like a good solution for us. If we can provide something like this to our users, it would help a lot.

[IgorTodorovskiIBM]

The toggle is documented here: https://github.com/ZOSOpenTools/gitport#encodings-and-zos-file-tags-ccsids . The latest release has the change: https://github.com/ZOSOpenTools/gitport/releases

[mfsysprog]

I don't feel that the latest fix has solved this issue. Tagging UTF-8 files (with zos-working-tree-encoding UTF-8) as ISO8859-1 is wrong in any case! For programs running in ASCII mode tagging as UTF-8 is fine with _BPXK_AUTOCVT=ON. It will not convert and this is exactly what you want in this case, it will still be readable and presented as UTF-8 for editors and tools that assume UTF-8 data. It will not convert to EBCDIC in this case, but this is also what you want. Tagging UTF-8 files as ISO8859-1 will allow for incorrect conversion with _BPXK_AUTOCVT=ON, as multibyte characters in the UTF-8 codepage will be converted to individual characters in EBCDIC which is incorrect. So in my opinion zos-working-tree-encoding should ALWAYS be respected and the resulting file should be tagged as such. If the user is willing to accept the risk of incorrect conversion from UTF-8 to ebcdic, they should run with _BPXK_AUTOCVT=ALL which implicitely accepts this risk.

[mfsysprog]

To clearify, this reasoning in the README.md is flawed in my opinion: Git on z/OS will do its best to associate a file tag (ccsid) with the git working-tree-encoding. However, there is a special case for UTF-8 encoded files. Such files are tagged as IS08859-1 (ccsid 819) because z/OS Open Tools currently acts on _BPXK_AUTOCVT=ON, which does not auto-convert files tagged with the UTF-8 tag (ccsid 1208).

Therefore, the default UTF-8 tag is IS08859-1 (or ccsid 819).

[DNikolaevAtRocket]

I don't feel that the latest fix has solved this issue. Tagging UTF-8 files (with zos-working-tree-encoding UTF-8) as ISO8859-1 is wrong in any case! For programs running in ASCII mode tagging as UTF-8 is fine with _BPXK_AUTOCVT=ON. It will not convert and this is exactly what you want in this case, it will still be readable and presented as UTF-8 for editors and tools that assume UTF-8 data. It will not convert to EBCDIC in this case, but this is also what you want. Tagging UTF-8 files as ISO8859-1 will allow for incorrect conversion with _BPXK_AUTOCVT=ON, as multibyte characters in the UTF-8 codepage will be converted to individual characters in EBCDIC which is incorrect. So in my opinion zos-working-tree-encoding should ALWAYS be respected and the resulting file should be tagged as such. If the user is willing to accept the risk of incorrect conversion from UTF-8 to ebcdic, they should run with _BPXK_AUTOCVT=ALL which implicitely accepts this risk.

I vote for the point. IMO, having UTF-8 tags on files specifically marked as UTF-8 is the right thing. That's what you expect when you set it via zos-working-tree-encoding. All the implications of how auto-conversion works is a different question; it's always pain anyways.

Also, we ran into another issue. We need a file being untagged when our repo is cloned. Is there a way to achieve it? ISO8859-1 text tags are always set by default. Pardon if it's off the topic.

[mfsysprog]

Can you explain why you would need it untagged? If there are specific usecases that require files to be untagged that is interesting.

[DNikolaevAtRocket]

We have a complex gradle build; a part of it (it's 3rd party Java code) shows exactly the behavior in your Java example above. Disabling autoconversion is not an option since it'd just break the rest of the build. One of the solutions that works for sure is to keep the affected files untagged.

[mfsysprog]

Ah ok, I did not create the java example, but I see the issue there. C Code compiled under unix system services can run with two pccsid's (proces codepages): 819 (ascii) or 1047(ebcdic) Enhanced ascii support uses this pccsid to see if it should convert file data. With _BPXK_AUTOCVT=ON it will convert if the file codepage (fccsid) does not match the pccsid, but only if it is a supported single byte characterset conversion. So if you tag a file ISO8859-1 and the pccsid is 1047 (like with Java), it will convert before the data reaches the JVM. If you tag a file as UTF-8 or leave it untagged, it will not be converted (it will copy binary and the data can then be interpreted inside the code like with the StandardCharsets.UTF_8 above).

So your usecase is probably that the data is actually ISO8859-1, but you don't want the filetag on it to prevent it from being converted to IBM-1047 before it is given to your gradle build. I guess it could be possible to have a zos-working-tree-encoding=untagged option and have it work the same as for binary data, but that would also mean that git will internally interpret the data as binary, which impacts the way tools like diff work if I'm correct.

[mfsysprog]

Have you ever tried to actually tag the data and run the gradle process with _BPXK_AUTOCVT=OFF?

[DNikolaevAtRocket]

Yes, the problematic Gradle step works with _BPXK_AUTOCVT=OFF (but other things like building native part fails terribly without auto-conversion - and there's no way we can re-work it).

So your usecase is probably that the data is actually ISO8859-1, but you don't want the filetag on it to prevent it from being converted to IBM-1047 before it is given to your gradle build. I guess it could be possible to have a zos-working-tree-encoding=untagged option and have it work the same as for binary data, but that would also mean that git will internally interpret the data as binary, which impacts the way tools like diff work if I'm correct.

Correct. On the other hand, I checked and apparently just binary tags work for our case. Not sure what new implications it can have compared to the untagged option.

Thanks!

[mfsysprog]

yes, basically everything that prevents automatic conversion would work in your case. Binary is the easiest option that git currently supports.

[IgorTodorovskiIBM]

Tagging UTF-8 files (with zos-working-tree-encoding UTF-8) as ISO8859-1 is wrong in any case! For programs running in ASCII mode tagging as UTF-8 is fine with _BPXK_AUTOCVT=ON. It will not convert and this is exactly what you want in this case, it will still be readable and presented as UTF-8 for editors and tools that assume UTF-8 data.

I think the issue was more with respect to _BPXK_AUTOCVT=ALL and there being data loss going from UTF-8 tagged files to the program ccsid ISO8859-1. @ccw-1 provided me with a sample that demonstrated this. @ccw-1 are you able to share?

Tagging UTF-8 files as ISO8859-1 will allow for incorrect conversion with _BPXK_AUTOCVT=ON, as multibyte characters in the UTF-8 codepage will be converted to individual characters in EBCDIC which is incorrect.

That is an interesting point. Do you know which characters fall into this category?

---

We decided to enforce tagging because the z/OS Open Tools rely on enhanced ASCII services which rely on file tagging.

Adding the binary attribute as @mfsysprog suggested will achieve the same effect: https://github.com/IgorTodorovskiIBM/EBCDICProject/blob/main/.gitattributes#L8

[mfsysprog]

Yes, _BPXK_AUTOCVT=ALL has to potential to convert incorrectly. Any character in the 8bit range (above decimal 127) has that potential. Utf8 needs the 8th bit to indicate that a second byte will follow (and if the next byte has the 8th bit on a third byte follows). The best example is perhaps the logical not character ¬. It is decimal 170 in ISO8859-1 and is encoded in UTF-8 as 0xC2AC. So if you have that character in the data and tag that file as ISO8859-1 it will show up as 2 characters, 0xC2 (an Â) and 0xAC (a ¢)

[mfsysprog]

I propose that we keep the latest fix, but make utf-8 the default tag for utf-8 files. That allows anybody who wants to change that to do so, but won't break the migration of everybody coming from the Rocket port of git.

[mfsysprog]

Does this version of git respect the zos-working-tree-encoding as introduced by the Rocket version of git?

[IgorTodorovskiIBM]

It's implemented slightly differently as [platform]-working-tree-encoding. On z/OS, [platform] = zos, so it's effectively the same besides the tagging behaviour for UTF-8 encoded files + disallowing untagged files.

[mfsysprog]

Ah ok, that's why I didn't find it

[mfsysprog]

I've tried to change the utf-8 default to 1208 and I see the issue now, all files that do not have a specific zos-working-tree-encoding set in the .gitattributes are tagged as utf-8. That seems undesirable, I can understand that default should be iso8859-1. However, the effect now is that files that are specifically tagged as utf-8 in zos-working-tree-encoding also become iso8859-1. That should not happen.

[IgorTodorovskiIBM]

Notice for Git Users: Upcoming Potentially Breaking Change After deliberating with the z/OS Open Tools dev community, and with the aim of facilitating a smoother transition from Rocket Software's Git port to the z/OS Open Tools Git, while adhering to the principle of least surprise, we have decided to eliminate the special case for UTF-8 encoded files.

Prior to this change, UTF-8 encoded git-managed files were tagged as ISO8859-1 (CCSID 819). They will now be tagged as UTF-8 (CCSID 1208). As a reminder, Git's internal encoding is UTF-8. Therefore, in the absence of a .gitattributes file, your files will be tagged as UTF-8 (CCSID 1208).

To maintain the existing behaviour, you can configure your git's utf8 ccsid setting to 819 using the following commands:

git config --global core.utf8ccsid 819 # Global setting, 819 represents the CCSID for the UTF8 file tag
git config core.utf8ccsid 819 # Local setting affecting the current repository

Alternatively, you can set the GIT_UTF8_CCSID environment variable:

export GIT_UTF8_CCSID=819 # Environment variable

To test the new behaviour, you can download the beta release here: https://github.com/ZOSOpenTools/gitport/releases/download/utf8ccsid/git-heads.v2.44.0.20240312_163022.zos.pax.Z

If there are no major concerns, we aim to make this release official next week.

[IgorTodorovskiIBM]

@nickrayjones the latest release has the above change, let me know if we can close this

[nickrayjones]

Yes. Thank you Igor, good to close