[Bug]: Can't run executables with CJK characters in the path if the bottle language is set to them.

Hairo commented 1 year ago

Describe the bug

If the bottle language is set to chinese, japanese, korean (and possibly others) launching an executable with CJK characters in the path throws this error in the console:

wine: failed to open "/home/hairo/Games/test/\xe3\x81\xa7\xe3\x81\x99\xe3\x81\xa7\xe3\x81\x99.exe": c0000135": c0000135

And the executable doesn't run.

If the bottle language is set to english (i tried with spanish too) the executables run as expected.

To Reproduce

Set the bottle language to chinese, japanese or korean
Launch an executable with CJK characters in the path/name
Nothing happens (executable doesn't run)

Package

Flatpak from Flathub

Distribution

EndeavourOS

Troubleshooting Logs

Version: 2022.10.14.1
Display:
    X.org: true
    X.org (port): :0
    Wayland: false
Graphics:
    vendors:
        amd:
            vendor: amd
            envs:
                DRI_PRIME: '1'
            icd: /usr/lib/x86_64-linux-gnu/GL/vulkan/icd.d/radeon_icd.x86_64.json:/usr/lib/i386-linux-gnu/GL/vulkan/icd.d/radeon_icd.i686.json
    prime:
        integrated: null
        discrete: null
Kernel:
    Type: Linux
    Version: 6.0.2-zen1-1-zen
Distro:
    Name: GNOME
    Version: '43'
Disk:
    Total: 16787501056
    Free: 16787361792
RAM:
    MemTotal: "33.6\_GB"
    MemAvailable: "29.2\_GB"
Bottles_envs: null

Display Protocol

X11

Display Backend used by Bottles

X11, I don't know

Additional context

I thought it was a locale issue but i do have ja_JP.utf8 enabled.

StoneMoe commented 1 year ago

The \xe3\x81\xa7\xe3\x81\x99\xe3\x81\xa7\xe3\x81\x99 hex string in filename is "ですです"(Japanese) in UTF-8 encoding.

Launch program from Bottles will pass language code without encoding suffix via LC_ALL according to these code: https://github.com/bottlesdevs/Bottles/blob/d04d791fd2faedb4a204f490c4b1e55d28153e2d/bottles/backend/wine/winecommand.py#L187-L190

https://github.com/bottlesdevs/Bottles/blob/d04d791fd2faedb4a204f490c4b1e55d28153e2d/bottles/backend/utils/manager.py#L302-L333

And system will choose the default one codeset for the specified language:

UTF-8 for es_ES(Spanish)
ISO-8859-1(aka latin-1) for en_US
GB18030 for zh_CN
EUC-JP for ja_JP

which usually is ASCII compatible(or not) for historic reason and be the first one in locale.gen AFAIK

> cat /etc/locale.gen | grep ja_JP
#ja_JP.EUC-JP EUC-JP  
#ja_JP.UTF-8 UTF-8

this MAYBE the cause of this problem, and a new LC_CTYPE encoding setting next to Language setting MAYBE a good solution.

More: https://www.gnu.org/software/libunistring/manual/html_node/Locale-encodings.html

latin-1 commented 1 year ago

For those looking for a workaround:

flatpak config --set extra-languages ja_JP.UTF-8
flatpak update

and set the environment variable in Bottles (rather than the language setting):

LANG=ja_JP.UTF-8

Note: Flatpak does not use the system's locale.gen, you must configure it manually. The .UTF-8 part appears to be required for the Bottles to function properly.

Revival8697 commented 1 year ago

@latin-1 this doesn't seem to work on my end. For example, can you install this EXE? https://tkool.jp/products/rtp/2000rtp.zip

trx1138 commented 1 year ago

@latin-1 this doesn't seem to work on my end. For example, can you install this EXE? https://tkool.jp/products/rtp/2000rtp.zip

installation failure could be irrelevant with locale config. try running explorer within your bottle and if it shows a cjk path in right encoding, that means there's nothing wrong with your locale configuration.

if your cjk path is showing in broken encoding, try running flatpak config --get extra-languages to check if it's configured properly.

Revival8697 commented 1 year ago

So I just tested the latest release and I discovered that it was the runner mishandling Japanese characters. The current default runner, soda-7.0-9 cannot handle the installation of this Japanese EXE and launching programs with Japanese characters in PATH/EXE. While wine-ge-proton7-37 handles them fine. This is not a Flatpak nor a Bottles bug. The runner is at fault here.

CC: @Hairo @mirkobrombin

Hairo commented 1 year ago

I've never used soda runners so it was probably something to do with the wine version. Just tried and yes, wine-ge-proton7-37 works.

kgraefe commented 2 weeks ago

The \xe3\x81\xa7\xe3\x81\x99\xe3\x81\xa7\xe3\x81\x99 hex string in filename is "ですです"(Japanese) in UTF-8 encoding.

Launch program from Bottles will pass language code without encoding suffix via LC_ALL according to these code:

https://github.com/bottlesdevs/Bottles/blob/d04d791fd2faedb4a204f490c4b1e55d28153e2d/bottles/backend/wine/winecommand.py#L187-L190

https://github.com/bottlesdevs/Bottles/blob/d04d791fd2faedb4a204f490c4b1e55d28153e2d/bottles/backend/utils/manager.py#L302-L333

And system will choose the default one codeset for the specified language:
UTF-8 for es_ES(Spanish)
ISO-8859-1(aka latin-1) for en_US
GB18030 for zh_CN
EUC-JP for ja_JP
which usually is ASCII compatible(or not) for historic reason and be the first one in locale.gen AFAIK
> cat /etc/locale.gen | grep ja_JP
#ja_JP.EUC-JP EUC-JP  
#ja_JP.UTF-8 UTF-8  
this MAYBE the cause of this problem, and a new LC_CTYPE encoding setting next to Language setting MAYBE a good solution.

More: https://www.gnu.org/software/libunistring/manual/html_node/Locale-encodings.html

I just ran into this. For taking a screenshot for https://github.com/bottlesdevs/programs/issues/339 I changed the bottle language to English instead of System (which is de_DE.UTF-8 on my system) and suddenly my program did not find its files anymore as their paths contained UTF-8 characters. So yes, there should be an option to set the encoding.

bottlesdevs / Bottles