VSCODE encoding error IBM-1208 since 3.1.0 update

[rsjrny]

Discussed in https://github.com/IBM/zopeneditor-about/discussions/330

^{Originally posted by **rsjrny** March 11, 2023} since the extension update 3.1.0 on 3/10/23 I have started receiving an encoding issue with CopyToPDS in groovy. If I initiate the same user build directly from USS the CopyToPDS works fine. I assume it has something to do with the transfer from windows to USS. I am at a loss as to where to start looking for a possible setting change. any help would be appreciated ``` 2023-03-11T11:06:46.416Z: Caught: com.ibm.dbb.build.BuildException: java.io.UnsupportedEncodingException: ibm-1208 2023-03-11T11:06:46.421Z: com.ibm.dbb.build.BuildException: java.io.UnsupportedEncodingException: ibm-1208 2023-03-11T11:06:46.426Z: at com.ibm.dbb.build.CopyToPDS.execute(CopyToPDS.java:89) 2023-03-11T11:06:46.429Z: at BuildUtilities.copySourceFiles(BuildUtilities.groovy:100) 2023-03-11T11:06:46.432Z: at Assembler$_run_closure1.doCall(Assembler.groovy:36) 2023-03-11T11:06:46.438Z: at Assembler.run(Assembler.groovy:28) 2023-03-11T11:06:46.443Z: at Assembler$run.callCurrent(Unknown Source) 2023-03-11T11:06:46.445Z: at com.ibm.dbb.groovy.ScriptLoader._run(ScriptLoader.groovy:124) 2023-03-11T11:06:46.448Z: at com.ibm.dbb.groovy.ScriptLoader$_run$0.call(Unknown Source) 2023-03-11T11:06:46.458Z: at com.ibm.dbb.groovy.ScriptLoader.runScript(ScriptLoader.groovy:81) 2023-03-11T11:06:46.461Z: at build$_run_closure1.doCall(build.groovy:63) 2023-03-11T11:06:46.464Z: at build.run(build.groovy:54) 2023-03-11T11:06:46.467Z: Caused by: java.io.UnsupportedEncodingException: ibm-1208 2023-03-11T11:06:46.472Z: at com.ibm.dbb.build.CopyToPDS.copyAsText(CopyToPDS.java:379) 2023-03-11T11:06:46.476Z: at com.ibm.dbb.build.CopyToPDS.copyAsText(CopyToPDS.java:306) 2023-03-11T11:06:46.478Z: at com.ibm.dbb.build.CopyToPDS.copyFile(CopyToPDS.java:232) 2023-03-11T11:06:46.481Z: at com.ibm.dbb.build.CopyToPDS.doCopy(CopyToPDS.java:179) 2023-03-11T11:06:46.490Z: at com.ibm.dbb.build.CopyToPDS.execute(CopyToPDS.java:85) ```

[phaumer]

It looks like that there is a regression that it will use utf-8 when no .gitattributes file is found instead of the encoding specified for the profile. As a workaround create a .gitattributes file with the encoding you need: https://www.ibm.com/docs/en/cloud-paks/z-modernization-stack/2022.4?topic=code-setting-up-user-build

[FALLAI-Denis]

Hi,

Same problem for me...

I do have a .gitattributes file, which states that by default files are encoded on the remote system in EBCDIC-1147; it does not specify anything for COBOL files (*.cbl) since it is the default that applies.

But, when I explicitly force the *.cbl and *.cpy patterns, it works!

# Les fichiers sont gérés et édités en UTF-8 ; sur z/OS, sauf contre indication, ils sont encodés en EBCDIC IBM-1147
*          text        working-tree-encoding=utf-8     zos-working-tree-encoding=ibm-1147
*.cbl      text        working-tree-encoding=utf-8     zos-working-tree-encoding=ibm-1147
*.cpy      text        working-tree-encoding=utf-8     zos-working-tree-encoding=ibm-1147
[...]

This log file is available at c:\Users\S0070188\AppData\Roaming\CodeMainframe\logs\20230313T214417\window1\exthost\IBM.zopeneditor\userbuild-20230313-204433.log
Welcome to IBM User Build. Run User Build from the pop-up menu in an open editor with a COBOL, PL/I, or HLASM program.
Visit our IBM Docs to learn about IBM User Build here: https://www.ibm.com/docs/en/z-modernization-stack/2022.1.1?topic=code-building-cobol-pli-hlasm-programs-user-build.
2023-03-13T21:07:22.812Z: =====================
2023-03-13T21:07:22.813Z: Running User Build...
2023-03-13T21:07:22.815Z: Identified the local root workspace path as c:\Trv\DepotsGIT\ceab3-environn-outinfr-central-mvp2
2023-03-13T21:07:22.817Z: ---------------------
2023-03-13T21:07:22.829Z: Checking dbb profile settings defined in your repository's zapp.yaml or zapp.json
2023-03-13T21:07:22.831Z: User setting dbbDefaultZappProfile: **dbb-userbuild**
2023-03-13T21:07:22.832Z: Will read settings from ZAPP profile called dbb-userbuild defined in User Settings
2023-03-13T21:07:22.834Z:     application: CEAB3.MVP2
    command: $DBB_HOME/bin/groovyz
    buildScriptArgs: --userBuild,--workspace /u/J070188/dbb,--application CEAB3.MVP2,--hlq J070188.DBB.CEAB3.MVP2,--outDir /u/J070188/dbb/logs
    buildScriptPath: /var/DBB/PROD/SCRIPTS/buildBPCE.groovy
    additionalDependencies: application-conf,.build,src\**\S9TL1A.cbl.build
    logFilePatterns: S9TL1A*.log,BuildReport.*
2023-03-13T21:07:22.836Z: ---------------------
2023-03-13T21:07:22.838Z: Checking user settings...
2023-03-13T21:07:22.840Z: User settings should be defined in your VS Code or Theia Settings under zopeneditor.userbuild.userSettings
2023-03-13T21:07:22.841Z:     dbbWorkspace: /u/J070188/dbb
    dbbHlq: J070188.DBB
    dbbLogDir: /u/J070188/dbb/logs
    dbbDefaultZappProfile: dbb-userbuild
    localLogDir: .reports
2023-03-13T21:07:22.844Z: Using .gitattributes file at c:\Trv\DepotsGIT\ceab3-environn-outinfr-central-mvp2\.gitattributes
2023-03-13T21:07:22.846Z: ---------------------
2023-03-13T21:07:22.848Z: The default Zowe CLI profile specified as TMVS in zopeneditor.zowe setting. Will try loading that.
2023-03-13T21:07:22.850Z: Using Zowe profile TMVS
2023-03-13T21:07:23.229Z: Using ssh profile TMVS-SSH
2023-03-13T21:07:23.232Z: 
2023-03-13T21:07:23.233Z: Checking if SSH profile TMVS-SSH can connect:
2023-03-13T21:07:23.804Z: Connected successfully.
2023-03-13T21:07:23.806Z: 
2023-03-13T21:07:24.033Z: Executing SSH command: 
mkdir -p '/u/J070188/dbb/CEAB3.MVP2/src/COBOL'
2023-03-13T21:07:24.516Z: $ mkdir -p '/u/J070188/dbb/CEAB3.MVP2/src/COBOL'
2023-03-13T21:07:24.518Z: $ exit $?
2023-03-13T21:07:24.524Z: Uploading file c:\Trv\DepotsGIT\ceab3-environn-outinfr-central-mvp2\src\COBOL\S9TL1A.cbl to /u/J070188/dbb/CEAB3.MVP2/src/COBOL/S9TL1A.cbl with encoding ibm-1147
2023-03-13T21:07:24.842Z: Using ssh profile TMVS-SSH
2023-03-13T21:07:24.844Z: Executing SSH command: 
chtag -tc ibm-1147 '/u/J070188/dbb/CEAB3.MVP2/src/COBOL/S9TL1A.cbl'
2023-03-13T21:07:25.354Z: $ chtag -tc ibm-1147 '/u/J070188/dbb/CEAB3.MVP2/src/COBOL/S9TL1A.cbl'
2023-03-13T21:07:25.369Z: $ exit $?
[...]

[FALLAI-Denis]

Hi,

@phaumer remember User Build - Priority for pattern management in .gitattributes

Thanks.

[phaumer]

@FALLAI-Denis and @rsjrny thanks. I can confirm two regressions (1) just using * does not work and (2) it will not use the Zowe profile default, but utf-8. We are fixing it. The workaround, as you suggested, is to provide a pattern with a file extension.

[FALLAI-Denis]

Hi @phaumer

The bug leads me to assume that the calculation of file attributes is provided internally by ZOE and does not rely on git's capabilities (in particular does not rely on the git check-attr command)...

If my analysis is correct, this is bound to cause problems... for example, it is possible in a Git Repository to define .gitattributes files at different levels of the folder hierarchy, and combines them to get the final attributes to apply to a resource, yet ZOE only seems to process only one .gitattributes file.

Git can also use techniques other than .gitattributes files: use of .git/info/attributes, or others.

From git-scm.com site:

_When deciding what attributes are assigned to a path, Git consults $GIT_DIR/info/attributes file (which has the highest precedence), .gitattributes file in the same directory as the path in question, and its parent directories up to the toplevel of the work tree (the further the directory that contains .gitattributes is from the path in question, the lower its precedence). Finally global and system-wide files are considered (they have the lowest precedence)._

When the .gitattributes file is missing from the work tree, the path in the index is used as a fall-back. During checkout process, .gitattributes in the index is used and then the file in the working tree is used as a fall-back.

_If you wish to affect only a single repository (i.e., to assign attributes to files that are particular to one user’s workflow for that repository), then attributes should be placed in the $GIT_DIR/info/attributes file. Attributes which should be version-controlled and distributed to other repositories (i.e., attributes of interest to all users) should go into .gitattributes files. Attributes that should affect all repositories for a single user should be placed in a file specified by thecore.attributesFile configuration option (see git-config](https://git-scm.com/docs/git-config) ). Its default value is $XDG_CONFIG_HOME/git/attributes. If $XDG_CONFIG_HOME is either not set or empty, $HOME/.config/git/attributes is used instead. Attributes for all users on a system should be placed in the $(prefix)/etc/gitattributes file._

For these reasons, I think ZOE should rely on a mechanism for defining attributes by Git itself, such as the git check-attr command.

[phaumer]

@FALLAI-Denis I do not disagree. Our original requirement was to use this file even when the user does not have git installed locally (e.g. they got the repo by downloading it as a zip from github), but I agree it might be worth supporting both type of checks.

[FALLAI-Denis]

Hi @phaumer,

In my opinion, if there is a .gitattributes file present in the Workspace, it means that we are probably in the presence of a Git Repository, and therefore that Git must be available in the working context of VS Code and therefore from Z Open Editor.

Moreover the IBM User Build function of Z Open Editor is mainly intended to work with zAppbuild, itself intended to work with IBM Dependency Based Build, and zAppbuild is still well built to work with Git, (and I even wonder if IBM DBB itself isn't built to work with Git, but that's less obvious).

So I think it's complicating your life to want Z Open Editor to do its own Git-style attribute management... Either Git is available and then you have to call the git check-attr command. Either Git is not available and then you have to use the encoding declared by the z/OSMF profile, or the zapp property groups for the RSE profile.

[phaumer]

We fixed the regressions for * and falling back to the Zowe profiles settings in case no match was found in v3.1.1 @rsjrny. Using the git command we treat as a separate enhancement request.

[rsjrny]

@phaumer I am still seeing the error after upgrading to Z Open Editor 3.1.1

2023-03-30T17:12:27.433Z: ** Build start at 20230330.011227.012 2023-03-30T17:12:28.113Z: Caught: java.io.UnsupportedEncodingException: ibm-1208 2023-03-30T17:12:28.115Z: java.io.UnsupportedEncodingException: ibm-1208 2023-03-30T17:12:28.116Z: at com.ibm.dbb.build.BuildProperties.loadProperties(BuildProperties.java:202) 2023-03-30T17:12:28.117Z: at com.ibm.dbb.build.BuildProperties.load(BuildProperties.java:161) 2023-03-30T17:12:28.118Z: at com.ibm.dbb.build.BuildProperties.load(BuildProperties.java:149) 2023-03-30T17:12:28.120Z: at build$_populateBuildProperties_closure4.doCall(build.groovy:376) 2023-03-30T17:12:28.121Z: at build.populateBuildProperties(build.groovy:371) 2023-03-30T17:12:28.124Z: at build.initializeBuildProcess(build.groovy:97) 2023-03-30T17:12:28.126Z: at build.run(build.groovy:32) 2023-03-30T17:12:28.921Z: ** Build finished

[phaumer]

@rsjrny I see. And is this still your .gitattributes file and it is located at the top-level directory of your VS Code workspace?

.gitattributes zos-working-tree-encoding=iso8859-1
.gitignore     zos-working-tree-encoding=iso8859-1
*              zos-working-tree-encoding=ibm-1047

which is what we tested with.

[rsjrny]

I had changed it to the following. It is in the top level .gitattributes zos-working-tree-encoding=iso8859-1 .gitignore zos-working-tree-encoding=iso8859-1 .c text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047 .h text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047 .maclib text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047 .samplib text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047 .srclib text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047 .prelink text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047 .log text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047 .cntlrep binary *.isptlib binary

[rsjrny]

whoops, forgot to copy the last line

* text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047

[rudyflores]

Hey @rsjrny ! When running a User Build with the following .gitattributes file you provided:

*              text working-tree-encoding=utf-8      zos-working-tree-encoding=ibm-1047
.gitattributes zos-working-tree-encoding=iso8859-1
.gitignore zos-working-tree-encoding=iso8859-1
*.c text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047
*.h text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047
*.maclib text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047
*.samplib text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047
*.srclib text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047
*.prelink text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047
*.log text working-tree-encoding=utf-8 zos-working-tree-encoding=ibm-1047
*.cntlrep binary
*.isptlib binary

What is the expected encoding? I see IBM-1208 is being applied which I assume is the default on your z/OS machine?

[rsjrny]

@phaumer

it is expecting IBM-1047 I issued ls -aLT thminit.srclib in the build directory and this is what I see t IBM-1047 T=on thminit.srclib

if I initiate the build on USS I have no issues. The problem only occurs when the build is initiated from VSCode

here is the vscode log

2023-03-31T16:43:18.580Z: Using .gitattributes file at c:\Repos\TMON\tmon-18926rl-tmhmv010\.gitattributes 2023-03-31T16:43:18.581Z: --------------------- 2023-03-31T16:43:18.585Z: The default Zowe CLI profile specified as RSI2 in zopeneditor.zowe setting. Will try loading that. 2023-03-31T16:43:18.587Z: Using Zowe profile RSI2 2023-03-31T16:43:18.971Z: Using ssh profile rsi2ssh 2023-03-31T16:43:18.972Z: 2023-03-31T16:43:18.973Z: Checking if SSH profile rsi2ssh can connect: 2023-03-31T16:43:19.484Z: Connected successfully. 2023-03-31T16:43:19.485Z: 2023-03-31T16:43:19.769Z: Executing SSH command: mkdir -p '/u/ts4579/build/tmhmv010' 2023-03-31T16:43:20.400Z: @@START OF COMMAND@@ 2023-03-31T16:43:20.402Z: $ mkdir -p '/u/ts4579/build/tmhmv010' 2023-03-31T16:43:20.411Z: Uploading file c:\Repos\TMON\tmon-18926rl-tmhmv010\thminit.srclib to /u/ts4579/build/tmhmv010/thminit.srclib with encoding ibm-1047 2023-03-31T16:43:20.820Z: Using ssh profile rsi2ssh 2023-03-31T16:43:20.822Z: Executing SSH command: chtag -tc ibm-1047 '/u/ts4579/build/tmhmv010/thminit.srclib' 2023-03-31T16:43:21.458Z: $ chtag -tc ibm-1047 '/u/ts4579/build/tmhmv010/thminit.srclib' 2023-03-31T16:43:21.480Z: Fetching build file dependencies... 2023-03-31T16:43:24.389Z: Found a total of 0 local include files: 2023-03-31T16:43:24.390Z: Checking if include files were updated since the last build... 2023-03-31T16:43:24.391Z: Updating 0/0 local include files. 2023-03-31T16:43:24.393Z: --------------------- 2023-03-31T16:43:24.697Z: Executing SSH command: $DBB_HOME/bin/groovyz /u/prodpc2/dbb-zappbuild-rsjrny/build.groovy --userBuild --workspace /u/ts4579/build/ --application tmhmv010 --hlq TMHMV010.U18926RL --outDir /u/ts4579/dbbuild/tmhmv010/logs --propFiles /u/ts4579/build/tmhmv010/procgrpList.properties --prodver tmhmv010 '/u/ts4579/build/tmhmv010/thminit.srclib' 2023-03-31T16:43:25.331Z: $ $DBB_HOME/bin/groovyz /u/prodpc2/dbb-zappbuild-rsjrny/build.groovy --use $ 2023-03-31T16:43:25.339Z: $ zappbuild-rsjrny/build.groovy --user <Build --workspace /u/ts4579/build $ 2023-03-31T16:43:25.343Z: $ serBuild --workspace /u/ts4579/build/ < --application tmhmv010 --hlq TMHMV0 $ 2023-03-31T16:43:25.348Z: $ ld/ --application tmhmv010 --hlq TMHMV01 <0.U18926RL --outDir /u/ts4579/dbbuil $ 2023-03-31T16:43:25.350Z: $ V010.U18926RL --outDir /u/ts4579/dbbuild </tmhmv010/logs --propFiles /u/ts4579 $ 2023-03-31T16:43:25.358Z: $ ild/tmhmv010/logs --propFiles /u/ts4579/ <build/tmhmv010/procgrpList.propertie $ 2023-03-31T16:43:25.361Z: $ 79/build/tmhmv010/procgrpList.properties < --prodver tmhmv010 '/u/ts4579/bu $ 2023-03-31T16:43:25.365Z: $ ies --prodver tmhmv010 '/u/ts4579/bui <ld/tmhmv010/thminit.srclib' 2023-03-31T16:43:32.267Z: ** Build start at 20230331.124332.043 2023-03-31T16:43:32.463Z: Caught: java.io.UnsupportedEncodingException: ibm-1208 2023-03-31T16:43:32.466Z: java.io.UnsupportedEncodingException: ibm-1208 2023-03-31T16:43:32.468Z: at com.ibm.dbb.build.BuildProperties.loadProperties(BuildProperties.java:202) 2023-03-31T16:43:32.470Z: at com.ibm.dbb.build.BuildProperties.load(BuildProperties.java:161) at com.ibm.dbb.build.BuildProperties.load(BuildProperties.java:149) 2023-03-31T16:43:32.472Z: at build$_populateBuildProperties_closure4.doCall(build.groovy:374) 2023-03-31T16:43:32.478Z: at build.populateBuildProperties(build.groovy:369) 2023-03-31T16:43:32.483Z: at build.initializeBuildProcess(build.groovy:94) 2023-03-31T16:43:32.486Z: at build.run(build.groovy:32) 2023-03-31T16:43:33.170Z: ** Build finished 2023-03-31T16:43:33.185Z: An error has occurred: Error: Something went wrong executing SSH command $DBB_HOME/bin/groovyz /u/prodpc2/dbb-zappbuild-rsjrny/build.groovy --userBuild --workspace /u/ts4579/build/ --application tmhmv010 --hlq TMHMV010.U18926RL --outDir /u/ts4579/dbbuild/tmhmv010/logs --propFiles /u/ts4579/build/tmhmv010/procgrpList.properties --prodver tmhmv010 '/u/ts4579/build/tmhmv010/thminit.srclib' 2023-03-31T16:43:33.187Z: Looking for user build logs in "/u/ts4579/dbbuild/tmhmv010/logs". 2023-03-31T16:43:33.189Z: Total log files found: 0 2023-03-31T16:43:33.190Z: Downloading logs to c:\Repos\TMON\tmon-18926rl-tmhmv010\logs 2023-03-31T16:43:33.191Z: Successfully finished downloading of 0 log files to c:\Repos\TMON\tmon-18926rl-tmhmv010\logs.

[FALLAI-Denis]

Hi @rsjrny ,

I analyzed your IBM User Build log: a priori the file thminit.srclib is transferred as IBM-1047 and tagged as IBM-1047.

2023-03-31T16:43:20.411Z: Uploading file c:\Repos\TMON\tmon-18926rl-tmhmv010\thminit.srclib to u/ts4579/build/tmhmv010/thminit.srclib with encoding ibm-1047

2023-03-31T16:43:21.458Z: $ chtag -tc ibm-1047 '/u/ts4579/build/tmhmv010/thminit.srclib'

The problem could come from another file already present in the user build workspace on USS...

2023-03-31T16:43:32.463Z: Caught: java.io.UnsupportedEncodingException: ibm-1208
2023-03-31T16:43:32.466Z: java.io.UnsupportedEncodingException: ibm-1208
2023-03-31T16:43:32.468Z: at com.ibm.dbb.build.BuildProperties.loadProperties(BuildProperties.java:202)
2023-03-31T16:43:32.470Z: at com.ibm.dbb.build.BuildProperties.load(BuildProperties.java:161)
                          at com.ibm.dbb.build.BuildProperties.load(BuildProperties.java:149)
2023-03-31T16:43:32.472Z: at build$_populateBuildProperties_closure4.doCall(build.groovy:374)
2023-03-31T16:43:32.478Z: at build.populateBuildProperties(build.groovy:369)
2023-03-31T16:43:32.483Z: at build.initializeBuildProcess(build.groovy:94)
2023-03-31T16:43:32.486Z: at build.run(build.groovy:32)

It could be a properties file: loadProperties function in Java exception message. Did you re-setup the user build workspace before starting the user build itself?

[rsjrny]

@FALLAI-Denis,

That was the issue, you nailed it. I deleted the build workspace, ran user build setup, then the userbuild.

all is well.

thanks!

[phaumer]

Great to hear. Thanks for the help @FALLAI-Denis.

[phaumer]

We added the ability to utilize the git check-attr command in the background to evaluate .gitattributes in Z Open Editor 3.2.0. If our code can find the command in the path it will use it, otherwise fall back to our previous algorithm.