apache / netbeans

Apache NetBeans
https://netbeans.apache.org/
Apache License 2.0
2.55k stars 813 forks source link

Nb21. Output encoding issue with Apache Tomcat out.log and Spring Boot out.log #7116

Open 6PATyCb opened 1 month ago

6PATyCb commented 1 month ago

Apache NetBeans version

Apache NetBeans 21

What happened

The issue happens in all my Java Maven projects when I try to output cyrillic to the console.

Language / Project Type / NetBeans Component

Java maven project

How to reproduce

Hello! Just create new project and send to log cyrrilic symbols. Look at screens: Nb20: изображение Nb21: springboot изображение tomcat изображение

Did this work correctly in an earlier version?

Apache NetBeans 20

Operating System

Windows 11

JDK

Liberica JDK 17x64

Apache NetBeans packaging

Apache NetBeans provided installer

Anything else

No response

Are you willing to submit a pull request?

No

mbien commented 1 month ago

strange, I can't remember any recent changes which could cause this. Was NetBeans running on the same JDK version for both screenshots?

this works for me on linux:

public class Mavenproject1 {
    public static void main(String[] args) {
        System.out.println(Runtime.version());
        System.out.println("кава");
    }
}

-> image

adding the windows label for now

neilcsmith-net commented 1 month ago

This sounds like the difference between running on JDK 17 and JDK 21 (well, 18+) to me. See #4771 and #4396 . Does setting the output encoding in the project or global Java settings help?

mbien commented 1 month ago

yeah that is why I asked about the JDK version. If it would be JDK 17 in both cases (as the issue says) it might be a new issue, otherwise it is the well known one you linked above.

6PATyCb commented 1 month ago

Nb21 изображение Nb20 изображение

6PATyCb commented 1 month ago

изображение изображение изображение изображение изображение

neilcsmith-net commented 1 month ago

OK, that somewhat rules out the issues linked to.

Can you try copying the launchers from NB20 into NB21? eg. bin/netbeans.exe and bin/netbeans64.exe.

I'm curious if some changes in the launcher code are causing this - eg. https://github.com/apache/netbeans-native-launchers/pull/7

6PATyCb commented 1 month ago

Nb21 works fine after replace изображение изображение

6PATyCb commented 1 month ago

wrong attach at previous post bin.zip

neilcsmith-net commented 1 month ago

Thanks for the confirmation. I'm curious if the released launchers for NB 21 work OK when running on JDK 21? I'm also curious if adding -J-Dstdout.encoding=UTF-8 in the default options in netbeans.conf will get the released launchers to work for you on JDK 17?

6PATyCb commented 1 month ago

It does not help. The issue is back изображение изображение

duoduobingbing commented 3 weeks ago

@6PATyCb It's interesting that this causes Tofu characters ⍰ (square boxes) instead of question marks (?). I can reproduce this when I intentionally mismatch the project's output encoding (which on JDK17 and lower is the same as the native encoding) and the one in Netbeans (which when run on JDK17 should also be the native encoding).

E.g. launching the Project with no-args on a West European locale with JDK 17 (output encoding=cp1252) and Netbeans 21 with (-Dstdout.encoding=utf-8)

Could you try the following?

Execute the following statements inside your project

        System.out.println("Пример✨️");
        System.out.println(Runtime.version());
        System.out.println("Charset.defaultCharset(): " + Charset.defaultCharset());
        System.out.println("file.encoding property:" + System.getProperty("file.encoding"));
        System.out.println("native.encoding property:" + System.getProperty("native.encoding"));
        System.out.println("sun.jnu.encoding property:" + System.getProperty("sun.jnu.encoding"));
        System.out.println("stdout.encoding property:" + System.getProperty("stdout.encoding"));
        System.out.println("sun.stdout.encoding property:" + System.getProperty("sun.stdout.encoding"));

        if (System.console() != null) {
            System.out.println("systemConsole encoding: " + System.console().charset());
        }

And then set -J-Dsun.stdout.encoding=..., -J-Dstdout.encoding=... and -J-Dfile.encoding=... to the output of the value of sun.jnu.encoding from the code block above.

Does this also solve your problem when using Netbeans 21 with JDK17 and the new launcher?


Another thing that would be interesting: When doing the above, does it change anything when you explicitly set a font under Tools>Options>Miscellaneous>Output>The '...' button next to the 'Font' that does support Cyrillic like Courier New?

6PATyCb commented 3 weeks ago

So, let try (Nb21, no changes in netbeans.conf): изображение

looks better after the changes: изображение


return to default: изображение изображение trying to change the font: изображение изображение with changes in netbeans.conf: изображение

duoduobingbing commented 3 weeks ago

@6PATyCb Thanks for testing. This proves that this is not a font or fontconfig.properties (You can configure the font fallbacks for differing encodings there individually) problem.

The problem is that the output encoding of Netbeans does not match the output encoding of the Java project when running everything (Java and Netbeans) with the default output encoding. For Windows the default Java output encoding is always the native codepage.

I can now reproduce this problem with the new launchers. It seems to make a difference whether you start bin/netbeans.exe or bin/netbeans64.exe.

Netbeans started from netbeans.exe with everything on default

When I start bin/netbeans.exe everything works correctly for me. The project's output encoding is cp1252 and Netbeans output encoding also seems to be cp1252

nb1

Netbeans shows Cp1252 in the about window and matches the Project's shown encoding Cp1252 resulting in the expected abcÄÖ✨️,Пример => abcÄÖ??,?????? where all characters not present in Cp1252 are turned into question marks.

Netbeans started from netbeans64.exe with everything on default

When I start bin/netbeans64.exe the output encodings become mismatched. The project's output encoding is cp1252 and Netbeans' output encoding seems to be utf-8

nb2

Netbeans shows UTF-8 in the about window and does not match the Project's shown encoding Cp1252 resulting in the unexpected abcÄÖ✨️,Пример => abc⍰⍰??,?????? where all characters not present in Cp1252 are turned into question marks and all characters differently encoded from unicode but present in Cp1252 turn into Tofu characters.

Workaround

The reporter in #7098 seemed to have the same issue. For now the best fix is to specify the encoding both on the project and inside etc/netbeans.conf to workaround those mismatches consistently.

For this I would recommend to set everything to UTF-8 as this has the added benefit of not being limited to one codepage.

The project's encoding can be set explicitly to UTF-8 with the JVM options -Dfile.encoding=utf-8 and -Dstdout.encoding=utf-8 to make the project use UTF-8 everywhere.

Luckily we can also set the Netbeans output encoding explicitly with with the -J-Dstdout.encoding=utf-8 and -J-Dfile.encoding=utf-8 options inside etc/netbeans.conf so that everything works all the time.

Adding both file.encoding and stdout.encoding makes this work consistently with Java 21 as well.

nb3

This yields the expected abcÄÖ✨️,Пример because both Netbeans' and the Project's output encoding is set to utf-8


Note: On Java 21 the 'about' dialogue is not of much help to determine the output encoding of Netbeans because it seems to display file.encoding which on JDK 21 is UTF-8 by default whereas stdout.encoding is still the native encoding e.g. win1252