What does `chcp` print on your console?

sormuras commented 9 months ago

For me it reads:

> chcp
Active code page: 65001

gmuecke commented 9 months ago

It reads

C:\Users\GeraldMücke>chcp
Aktive Codepage: 850.

sormuras commented 9 months ago

Could you please (temporarily) activate "Beta: Use Unicode UTF-8 for worldwide language support"...

...and retry to run the java launcher?

gmuecke commented 9 months ago

I did

C:\Users\GeraldMücke>chcp
Aktive Codepage: 65001

now run the launcher:

D:\git\umlaut_argfile_reproducer>type argfile.txt
-classpath target/ümlaut_demö.jar
org.example.Main

D:\git\umlaut_argfile_reproducer>java @argfile.txt
Fehler: Hauptklasse org.example.Main konnte nicht gefunden oder geladen werden
Ursache: java.lang.ClassNotFoundException: org.example.Main

D:\git\umlaut_argfile_reproducer>java -classpath target/ümlaut_demö.jar org.example.Main
Fehler: Hauptklasse org.example.Main konnte nicht gefunden oder geladen werden
Ursache: java.lang.ClassNotFoundException: org.example.Main

at least the bevahior with and without argfile is now consistent :)

sormuras commented 9 months ago

That's consistent - and odd. As it works on my machine, following the repro steps from the README:

J:\github\gmuecke\umlaut_argfile_reproducer>type argfile.txt
-classpath target/ümlaut_demö.jar
org.example.Main

J:\github\gmuecke\umlaut_argfile_reproducer>java @argfile.txt
Hello world!

J:\github\gmuecke\umlaut_argfile_reproducer>java -classpath target/ümlaut_demö.jar org.example.Main
Hello world!

What happens on your machine after a fresh checkout into a new directory?

sormuras commented 9 months ago

Also works as expected on a GitHub Codespace machine (some Linux variant, I suppose):

@sormuras ➜ /workspaces/umlaut_argfile_reproducer (main) $ bash demo.sh 
Hello world!

gmuecke commented 9 months ago

same result, I did

D:\>chcp
Aktive Codepage: 65001.

D:\>mkdir test

D:\>cd test

D:\test>git clone https://github.com/gmuecke/umlaut_argfile_reproducer.git
Cloning into 'umlaut_argfile_reproducer'...
remote: Enumerating objects: 21, done.
remote: Counting objects: 100% (21/21), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 21 (delta 2), reused 21 (delta 2), pack-reused 0
Receiving objects: 100% (21/21), done.
Resolving deltas: 100% (2/2), done.

D:\test>cd umlaut_argfile_reproducer

D:\test\umlaut_argfile_reproducer>mvn package
[INFO] Scanning for projects...
[INFO]
[INFO] ---------------< org.example:umlaut_argfile_reproducer >----------------
[INFO] Building umlaut_argfile_reproducer 1.0-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ umlaut_argfile_reproducer ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\test\umlaut_argfile_reproducer\src\main\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ umlaut_argfile_reproducer ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to D:\test\umlaut_argfile_reproducer\target\classes
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ umlaut_argfile_reproducer ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\test\umlaut_argfile_reproducer\src\test\resources
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ umlaut_argfile_reproducer ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ umlaut_argfile_reproducer ---
[INFO] No tests to run.
[INFO]
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ umlaut_argfile_reproducer ---
[INFO] Building jar: D:\test\umlaut_argfile_reproducer\target\ümlaut_demö.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  2.330 s
[INFO] Finished at: 2023-12-07T09:48:12+01:00
[INFO] ------------------------------------------------------------------------

D:\test\umlaut_argfile_reproducer>demo.cmd

D:\test\umlaut_argfile_reproducer>java @argfile.txt
Fehler: Hauptklasse org.example.Main konnte nicht gefunden oder geladen werden
Ursache: java.lang.ClassNotFoundException: org.example.Main

D:\test\umlaut_argfile_reproducer>java -classpath target\ümlaut_demö.jar org.example.Main
Fehler: Hauptklasse org.example.Main konnte nicht gefunden oder geladen werden
Ursache: java.lang.ClassNotFoundException: org.example.Main

sormuras commented 9 months ago

Please re-run via java -XshowSettings:properties @argfile.txt - I suspect some odd encodings being set...

gmuecke commented 9 months ago

silly me, I had run the above attempt (with fresh checkout etc) with my java_home still set to jdk17 (my system default). I reattempted with jdk21 - et voilà - it's working now:

D:\test\umlaut_argfile_reproducer>java @argfile.txt
Hello world!

so its:

jdk17 (17.0.3+7) + system codepage 65001: not working
jdk21 (21.0.1+12) + system codepage 65001: working

so I will revert my system default code page to 850 now and will retest again with both JDKs

sormuras commented 9 months ago

After setting 8 as source and target in the pom.xml, I retried with Java OpenJDK builds 11, 17, and 21. All fine here.

gmuecke commented 9 months ago

and back with system settings to default (codepage 850):

D:\>cd test\umlaut_argfile_reproducer

D:\test\umlaut_argfile_reproducer>set JAVA_HOME=C:\Java\temurin-21.0.1

D:\test\umlaut_argfile_reproducer>set PATH=%JAVA_HOME%\bin;%PATH%

D:\test\umlaut_argfile_reproducer>java -version
openjdk version "21.0.1" 2023-10-17 LTS
OpenJDK Runtime Environment Temurin-21.0.1+12 (build 21.0.1+12-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.1+12 (build 21.0.1+12-LTS, mixed mode, sharing)

D:\test\umlaut_argfile_reproducer>chcp
Aktive Codepage: 850.

D:\test\umlaut_argfile_reproducer>demo.cmd

D:\test\umlaut_argfile_reproducer>java @argfile.txt
Fehler: Hauptklasse org.example.Main konnte nicht gefunden oder geladen werden
Ursache: java.lang.ClassNotFoundException: org.example.Main

D:\test\umlaut_argfile_reproducer>chcp 65001
Aktive Codepage: 65001.

D:\test\umlaut_argfile_reproducer>demo.cmd

D:\test\umlaut_argfile_reproducer>java @argfile.txt
Fehler: Hauptklasse org.example.Main konnte nicht gefunden oder geladen werden
Ursache: java.lang.ClassNotFoundException: org.example.Main

jdk17 (17.0.3+7) + system codepage 850: not working
jdk21 (21.0.1+12) + system codepage 850: not working

So it seems the system setting has indeed an effect

sormuras commented 9 months ago

Please run with -XshowSettings:properties and list the encoding-related pairs here. Thanks.

For me, running Java 21 and with "Beta: Use Unicode UTF-8..." enabled, they are:

    file.encoding = UTF-8
    native.encoding = UTF-8
    stderr.encoding = UTF-8
    stdout.encoding = UTF-8
    sun.jnu.encoding = UTF-8

gmuecke commented 9 months ago

jdk21, codepage 850

D:\test\umlaut_argfile_reproducer>java -XshowSettings:properties @argfile.txt
Property settings:
    file.encoding = UTF-8
    file.separator = \
    java.class.path = target/Ã¼mlaut_demÃ¶.jar
    java.class.version = 65.0
    java.home = C:\Java\temurin-21.0.1
    java.io.tmpdir = C:\Users\GERALD~2\AppData\Local\Temp\
    java.library.path = C:\Java\temurin-21.0.1\bin
        C:\WINDOWS\Sun\Java\bin
        C:\WINDOWS\system32
        C:\WINDOWS
        C:\Java\temurin-21.0.1\bin
        C:\Windows\system32
        C:\Windows
        C:\Windows\System32\Wbem
        C:\Windows\System32\WindowsPowerShell\v1.0\
        C:\Windows\System32\OpenSSH\
        C:\Program Files\PuTTY\
        C:\Program Files\Amazon\AWSCLIV2\
        C:\ProgramData\chocolatey\bin
        C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR
        C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common
        C:\WINDOWS\system32
        C:\WINDOWS
        C:\WINDOWS\System32\Wbem
        C:\WINDOWS\System32\WindowsPowerShell\v1.0\
        C:\WINDOWS\System32\OpenSSH\
        C:\Program Files\Git\cmd
        C:\Program Files\Docker\Docker\resources\bin
        C:\Program Files\dotnet\
        C:\Users\GeraldMücke\.jbang\bin
        C:\Users\GeraldMücke\AppData\Local\Microsoft\WindowsApps
        C:\Java\jdk-17.0.3+7\bin
        C:\Users\GeraldMücke\go\bin
        C:\tools\gohugo
        C:\scala\scala-2.13.4\bin
        C:\Go\bin
        C:\Python\Python39\\Scripts\
        C:\Python\Python39\
        C:\Node\node-v16.15.0-win-x64
        D:\repository\npm
        D:\tools\apache-maven-3.8.5\bin
        D:\tools\gradle-7.5.1\bin
        C:\Users\GeraldMücke\AppData\Local\JetBrains\Toolbox\scripts
        C:\Users\GeraldMücke\AppData\Local\Microsoft\WindowsApps

        .
    java.runtime.name = OpenJDK Runtime Environment
    java.runtime.version = 21.0.1+12-LTS
    java.specification.name = Java Platform API Specification
    java.specification.vendor = Oracle Corporation
    java.specification.version = 21
    java.vendor = Eclipse Adoptium
    java.vendor.url = https://adoptium.net/
    java.vendor.url.bug = https://github.com/adoptium/adoptium-support/issues
    java.vendor.version = Temurin-21.0.1+12
    java.version = 21.0.1
    java.version.date = 2023-10-17
    java.vm.compressedOopsMode = Zero based
    java.vm.info = mixed mode, sharing
    java.vm.name = OpenJDK 64-Bit Server VM
    java.vm.specification.name = Java Virtual Machine Specification
    java.vm.specification.vendor = Oracle Corporation
    java.vm.specification.version = 21
    java.vm.vendor = Eclipse Adoptium
    java.vm.version = 21.0.1+12-LTS
    jdk.debug = release
    line.separator = \r \n
    native.encoding = Cp1252
    os.arch = amd64
    os.name = Windows 11
    os.version = 10.0
    path.separator = ;
    stderr.encoding = UTF-8
    stdout.encoding = UTF-8
    sun.arch.data.model = 64
    sun.boot.library.path = C:\Java\temurin-21.0.1\bin
    sun.cpu.endian = little
    sun.cpu.isalist = amd64
    sun.io.unicode.encoding = UnicodeLittle
    sun.java.command = org.example.Main
    sun.java.launcher = SUN_STANDARD
    sun.jnu.encoding = Cp1252
    sun.management.compiler = HotSpot 64-Bit Tiered Compilers
    sun.os.patch.level =
    user.country = CH
    user.dir = D:\test\umlaut_argfile_reproducer
    user.home = C:\Users\GeraldMücke
    user.language = de
    user.name = GeraldMücke
    user.script =
    user.variant =

Fehler: Hauptklasse org.example.Main konnte nicht gefunden oder geladen werden
Ursache: java.lang.ClassNotFoundException: org.example.Main

jdk17, codepage 850

D:\test\umlaut_argfile_reproducer>java -XshowSettings:properties @argfile.txt
Property settings:
    file.encoding = Cp1252
    file.separator = \
    java.class.path = target/Ã¼mlaut_demÃ¶.jar
    java.class.version = 61.0
    java.home = C:\Java\temurin-17.0.9
    java.io.tmpdir = C:\Users\GERALD~2\AppData\Local\Temp\
    java.library.path = C:\Java\temurin-17.0.9\bin
        C:\WINDOWS\Sun\Java\bin
        C:\WINDOWS\system32
        C:\WINDOWS
        C:\Windows\system32
        C:\Windows
        C:\Windows\System32\Wbem
        C:\Windows\System32\WindowsPowerShell\v1.0\
        C:\Windows\System32\OpenSSH\
        C:\Program Files\PuTTY\
        C:\Program Files\Amazon\AWSCLIV2\
        C:\ProgramData\chocolatey\bin
        C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR
        C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common
        C:\WINDOWS\system32
        C:\WINDOWS
        C:\WINDOWS\System32\Wbem
        C:\WINDOWS\System32\WindowsPowerShell\v1.0\
        C:\WINDOWS\System32\OpenSSH\
        C:\Program Files\Git\cmd
        C:\Program Files\Docker\Docker\resources\bin
        C:\Program Files\dotnet\
        C:\Users\GeraldMücke\.jbang\bin
        C:\Users\GeraldMücke\AppData\Local\Microsoft\WindowsApps
        C:\Java\jdk-17.0.3+7\bin
        C:\Users\GeraldMücke\go\bin
        C:\tools\gohugo
        C:\scala\scala-2.13.4\bin
        C:\Go\bin
        C:\Python\Python39\\Scripts\
        C:\Python\Python39\
        C:\Node\node-v16.15.0-win-x64
        D:\repository\npm
        D:\tools\apache-maven-3.8.5\bin
        D:\tools\gradle-7.5.1\bin
        C:\Users\GeraldMücke\AppData\Local\JetBrains\Toolbox\scripts
        C:\Users\GeraldMücke\AppData\Local\Microsoft\WindowsApps

        .
    java.runtime.name = OpenJDK Runtime Environment
    java.runtime.version = 17.0.9+9
    java.specification.name = Java Platform API Specification
    java.specification.vendor = Oracle Corporation
    java.specification.version = 17
    java.vendor = Eclipse Adoptium
    java.vendor.url = https://adoptium.net/
    java.vendor.url.bug = https://github.com/adoptium/adoptium-support/issues
    java.vendor.version = Temurin-17.0.9+9
    java.version = 17.0.9
    java.version.date = 2023-10-17
    java.vm.compressedOopsMode = Zero based
    java.vm.info = mixed mode, sharing
    java.vm.name = OpenJDK 64-Bit Server VM
    java.vm.specification.name = Java Virtual Machine Specification
    java.vm.specification.vendor = Oracle Corporation
    java.vm.specification.version = 17
    java.vm.vendor = Eclipse Adoptium
    java.vm.version = 17.0.9+9
    jdk.debug = release
    line.separator = \r \n
    native.encoding = Cp1252
    os.arch = amd64
    os.name = Windows 11
    os.version = 10.0
    path.separator = ;
    sun.arch.data.model = 64
    sun.boot.library.path = C:\Java\temurin-17.0.9\bin
    sun.cpu.endian = little
    sun.cpu.isalist = amd64
    sun.io.unicode.encoding = UnicodeLittle
    sun.java.command = org.example.Main
    sun.java.launcher = SUN_STANDARD
    sun.jnu.encoding = Cp1252
    sun.management.compiler = HotSpot 64-Bit Tiered Compilers
    sun.os.patch.level =
    sun.stderr.encoding = UTF-8
    sun.stdout.encoding = UTF-8
    user.country = CH
    user.dir = D:\test\umlaut_argfile_reproducer
    user.home = C:\Users\GeraldMücke
    user.language = de
    user.name = GeraldMücke
    user.script =
    user.variant =

Fehler: Hauptklasse org.example.Main konnte nicht gefunden oder geladen werden
Ursache: java.lang.ClassNotFoundException: org.example.Main

the java.class.path looks odd

sormuras commented 9 months ago

the java.class.path looks odd

java.class.path = target/Ã¼mlaut_demÃ¶.jar

Yeah. That's due to the java launcher using the "system default encoding" (cp1252) to read the argument file. And that encoding is not "ASCII friendly".

Mind those two facts as a general background:

java argfiles have a strict encoding rule: "The argument file must contain only ASCII characters or characters in system default encoding that's ASCII friendly, such as UTF-8." - more details at
- Java 9: https://docs.oracle.com/javase/9/tools/java.htm#JSWOR-GUID-4856361B-8BFD-4964-AE84-121F5F6CF111
- Java 21: https://docs.oracle.com/en/java/javase/21/docs/specs/man/java.html#java-command-line-argument-files
Since Java 18, Java uses "UTF-8 by Default" - more details at https://openjdk.org/jeps/400

gmuecke commented 9 months ago

I see, setting aside, that ASCII itself is not very umlaut-friendly (and I'm not sure, whether I should blame MS for using my umlaut-friendly name as profile path, because java is not the only program I have issues with this) - what is your suggestion now?

I assume, it's rather not a direct bug in jdk, but a nasty combination of conditions (windows default encoding not ascii friendly, java 17 not yet utf-8 default, windows blatantly uses umlauts in profile paths and intellj also just uses defaults). OTOH, the Windows UTF-8 support is still beta and not the default, so there is still a non-zero chance there are some users with special characters in their profile paths that still have a ascii-unfriendly codepage as their system default.

So I guess, my best bet for the moment is to have the Windows utf-8 support enabled (and pray the beta doesn't lead to any unforeseen effects), try to bump all the java projects to >= jdk21 (currently I'm still on gradle 7 which doesn't support jdk21) or rename my profile path

First wörld problems...

sormuras commented 9 months ago

Perhaps @brunoborges has some insight here? See https://github.com/microsoft/java-wdb/issues/13 for some links and other issues in the microsoft/java-wdb projects.

sormuras commented 9 months ago

I turned on Windows' UTF-8 support on my machine for time now (years?) and didn't have any real issues so far. Turning it off leads to more issues with more application, to my experience. So, let's use UFT-8 everywhere.

Same goes for the JDK version: ~~JDK-8~~ ... erm, JDK 21! This is the way.

In addition, it would be good to file an issue with IDEA's tracker to either use pure ASCII encoding or the system encoding when writing argument files on behalf of the user (and warn or even break if the system encoding is not ASCII-friendly). Do you want to create an issue at https://youtrack.jetbrains.com/issues/IDEA ?

sormuras commented 9 months ago

Oh, I tend to close https://bugs.openjdk.org/browse/JDK-8321268 as "works-as-designed".

gmuecke / umlaut_argfile_reproducer

What does `chcp` print on your console? #1