search

lutris / lutris

Lutris desktop client
https://lutris.net
GNU General Public License v3.0
7.8k stars 686 forks source link

lutris can't handle install script with non-ascii filename #5571

Closed shiluotang closed 1 month ago

shiluotang commented 1 month ago

Bug description

lutris can't handle non ascii filename in install script

it give following information on console.

Traceback (most recent call last): File "/usr/share/lutris/bin/lutris-wrapper", line 202, in main() File "/usr/share/lutris/bin/lutris-wrapper", line 172, in main while watcher.is_alive(): ^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lutris/util/process_watcher.py", line 75, in is_alive return next(self.iterate_processes(), None) is not None ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lutris/util/process_watcher.py", line 65, in iterate_processes if child.state == 'Z': ^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lutris/util/process.py", line 91, in state _stat = self.get_stat() ^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/lutris/util/process.py", line 48, in get_stat _stat = stat_file.readline() ^^^^^^^^^^^^^^^^^^^^ File "", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 21-22: invalid continuation byte

How to Reproduce

Steps to reproduce the behavior:

  1. write a yml script for game installation, but installer filename is non-ascii (e.g. "1.07——威震八方.exe")
  2. add game from local script

More to explain this problem

  1. write a simple program which will running forever, compile it with gcc , and name it as "1.07--威震八方.exe" or "1.07——威震八方.exe"
  2. run this program and find its pid
  3. write a simple python script use the same logic of get_stat in "process.py" to read /proc/{pid}/stat content

program to provide non ascii filename

#include <stdlib.h>
#include <stdio.h>
#include <limits.h>

#include <unistd.h>

int main(int argc, char* *argv) {
    for (int i = 0; i < INT_MAX; ++i) {
        printf("sleep for 1 second\n");
        sleep(1);
    }
    return EXIT_SUCCESS;
}

python script to check process state as the process.py does

#!/usr/bin/env python3

import io
import sys

if __name__ == "__main__":
    filename = "/proc/%s/stat" % (sys.argv[1])
    with open(filename, encoding='utf-8') as f:
        print(f.readline())

Expected behavior

lutris can handle installer script with non ascii installer filename

There're some reasons of this problem

  1. /proc/{pid}/stat content is not good at non-ascii program filename. This can be verified with simple programs
  2. proccess.py use /proc/{pid}/stat for lazy, it not a good implementation. every filename in /proc/{pid}/stat is limited, about 15 characters on debian 12, not to mention the non-ascii encoding problem
  3. process.py use 'utf-8' for encoding silver bullet , but real world is not that easy.

Possible solution

  1. use /proc/{pid}/status which also has alpha status indicator
  2. open proc stat or status file with binary mode, skip name item, just lookup for status
  3. if filename is important, try to use /proc/{pid}/cmdline, it has no encoding problem on debian 12
  4. readlink with /proc/{pid}/exe will also give filename

Log output

DEBUG    2024-08-19 08:14:08,126 [interpreter._iter_commands:322]:Installer command: {'insert-disc': {'message': '请挂载流星蝴蝶升级补丁.iso', 'requires': 'install-packages/1.07——威震八方.exe'}}
DEBUG    2024-08-19 08:14:14,361 [commands._find_matching_disc:257]:Found install-packages/1.07——威震八方.exe on cdrom /media/samuel/流星蝴蝶 剑升级补丁
DEBUG    2024-08-19 08:14:14,362 [interpreter._iter_commands:322]:Installer command: {'task': {'executable': '$DISC/install-packages/1.07——威 震八方.exe', 'name': 'wineexec', 'prefix': '$GAMEDIR'}}
DEBUG    2024-08-19 08:14:14,384 [wine.use_lutris_runtime:121]:/home/samuel/.local/share/lutris/runners/wine/wine-ge-8-26-x86_64/bin/wine is provided by Lutris, using runtime
WARNING  2024-08-19 08:14:14,387 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,387 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,387 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,387 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,387 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,387 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,387 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,387 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,387 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,387 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,387 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,412 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,413 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,439 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,439 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,439 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,439 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,439 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,442 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,442 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,442 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,442 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,442 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,442 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,442 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,442 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,442 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,442 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,443 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,443 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,443 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,443 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,443 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,445 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,445 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,445 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,445 [runner.game_config:86]:Accessing game config while runner wasn't given one.
WARNING  2024-08-19 08:14:14,445 [runner.game_config:86]:Accessing game config while runner wasn't given one.
DEBUG    2024-08-19 08:14:14,450 [command.start:141]:__GL_SHADER_DISK_CACHE="1"
DEBUG    2024-08-19 08:14:14,450 [command.start:141]:__GL_SHADER_DISK_CACHE_PATH="/home/samuel/.cache/lutris/shaders/wine"
DEBUG    2024-08-19 08:14:14,450 [command.start:141]:DRI_PRIME="1"
DEBUG    2024-08-19 08:14:14,450 [command.start:141]:VK_ICD_FILENAMES="/usr/share/vulkan/icd.d/nvidia_icd.json"
DEBUG    2024-08-19 08:14:14,450 [command.start:141]:LD_LIBRARY_PATH="/home/samuel/.local/share/lutris/runners/wine/wine-ge-8-26-x86_64/lib:/home/samuel/.local/share/lutris/runners/wine/wine-ge-8-26-x86_64/lib64:/lib/x86_64-linux-gnu:/lib/i386-linux-gnu:/lib32:/lib:/lib/i386-linux-gnu/i686/sse2:/usr/lib/x86_64-linux-gnu/libfakeroot:/lib64:/lib64:/usr/lib:/usr/lib64:/usr/lib32:/usr/lib64:/usr/lib/i386-linux-gnu:/usr/lib/x86_64-linux-gnu:/home/samuel/.local/share/lutris/runtime/Ubuntu-18.04-i686:/home/samuel/.local/share/lutris/runtime/steam/i386/lib/i386-linux-gnu:/home/samuel/.local/share/lutris/runtime/steam/i386/lib:/home/samuel/.local/share/lutris/runtime/steam/i386/usr/lib/i386-linux-gnu:/home/samuel/.local/share/lutris/runtime/steam/i386/usr/lib:/home/samuel/.local/share/lutris/runtime/Ubuntu-18.04-x86_64:/home/samuel/.local/share/lutris/runtime/steam/amd64/lib/x86_64-linux-gnu:/home/samuel/.local/share/lutris/runtime/steam/amd64/lib:/home/samuel/.local/share/lutris/runtime/steam/amd64/usr/lib/x86_64-linux-gnu:/home/samuel/.local/share/lutris/runtime/steam/amd64/usr/lib"
DEBUG    2024-08-19 08:14:14,450 [command.start:141]:DXVK_HUD="devinfo,fps"
DEBUG    2024-08-19 08:14:14,450 [command.start:141]:WINEDEBUG="-all"
DEBUG    2024-08-19 08:14:14,450 [command.start:141]:DXVK_LOG_LEVEL="none"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:WINEARCH="win64"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:WINE="/home/samuel/.local/share/lutris/runners/wine/wine-ge-8-26-x86_64/bin/wine"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:WINE_MONO_CACHE_DIR="/home/samuel/.local/share/lutris/runners/wine/lutris-fshack-7.2-x86_64/mono"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:WINE_GECKO_CACHE_DIR="/home/samuel/.local/share/lutris/runners/wine/lutris-fshack-7.2-x86_64/gecko"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:GST_PLUGIN_SYSTEM_PATH_1_0="/home/samuel/.local/share/lutris/runners/wine/lutris-fshack-7.2-x86_64/lib64/gstreamer-1.0/:/home/samuel/.local/share/lutris/runners/wine/lutris-fshack-7.2-x86_64/lib/gstreamer-1.0/"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:WINEPREFIX="/home/samuel/Games/meteor-butterfly-and-a-blade"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:WINEESYNC="1"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:WINEFSYNC="1"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:WINE_FULLSCREEN_FSR="1"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:DXVK_NVAPIHACK="0"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:DXVK_ENABLE_NVAPI="1"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:PROTON_EAC_RUNTIME="/home/samuel/.local/share/lutris/runtime/eac_runtime"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:WINEDLLOVERRIDES="winemenubuilder="
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:LANG="zh_CN.UTF-8"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:PYTHONPATH="/usr/lib/lutris:/usr/games:/usr/lib/python311.zip:/usr/lib/python3.11:/usr/lib/python3.11/lib-dynload:/usr/lib/python3/dist-packages"
DEBUG    2024-08-19 08:14:14,451 [command.start:141]:LUTRIS_GAME_UUID="39ec9afa-0e37-4572-b307-ac4d721a62ac"
Started initial process 465291 from /home/samuel/.local/share/lutris/runners/wine/wine-ge-8-26-x86_64/bin/wine /media/samuel/流星蝴蝶剑升级补 丁/install-packages/1.07——威震八方.exe
Start monitoring process.
fsync: up and running.
wine: RLIMIT_NICE is <= 20, unable to use setpriority safely
Traceback (most recent call last):
  File "/usr/share/lutris/bin/lutris-wrapper", line 202, in <module>
    main()
  File "/usr/share/lutris/bin/lutris-wrapper", line 172, in main
    while watcher.is_alive():
          ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/lutris/util/process_watcher.py", line 75, in is_alive
    return next(self.iterate_processes(), None) is not None
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/lutris/util/process_watcher.py", line 65, in iterate_processes
    if child.state == 'Z':
       ^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/lutris/util/process.py", line 91, in state
    _stat = self.get_stat()
            ^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/lutris/util/process.py", line 48, in get_stat
    _stat = stat_file.readline()
            ^^^^^^^^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 21-22: invalid continuation byte
WARNING  2024-08-19 08:14:14,894 [command.get_return_code:185]:No file /tmp/lutris-39ec9afa-0e37-4572-b307-ac4d721a62ac
DEBUG    2024-08-19 08:14:14,894 [command.on_stop:195]:Process 465289 has terminated with code
DEBUG    2024-08-19 08:14:15,452 [commands._monitor_task:454]:Return code:
ERROR    2024-08-19 08:14:15,452 [errors.__init__:18]:Command exited with code
None
INFO     2024-08-19 08:14:34,106 [interpreter.revert:385]:Cancelling installation of 流星蝴蝶剑.net
DEBUG    2024-08-19 08:14:34,109 [wine.winekill:172]:Killing all wine processes: ['/home/samuel/.local/share/lutris/runners/wine/wine-ge-8-26-x86_64/bin/wineserver', '-k']
DEBUG    2024-08-19 08:14:34,109 [wine.winekill:173]:   Wine prefix: /home/samuel/Games/meteor-butterfly-and-a-blade
DEBUG    2024-08-19 08:14:34,110 [wine.winekill:174]:   Wine arch: win64
DEBUG    2024-08-19 08:14:34,110 [wine.winekill:180]:Waiting for wine processes to terminate
DEBUG    2024-08-19 08:14:34,111 [wine.winekill:196]:Done waiting.
DEBUG    2024-08-19 08:14:34,111 [system.remove_folder:291]:Removing folder /home/samuel/Games/meteor-butterfly-and-a-blade
DEBUG    2024-08-19 08:14:34,226 [system.remove_folder:291]:Removing folder /home/samuel/.cache/lutris/installer/meteor-butterfly-and-a-blade
DEBUG    2024-08-19 08:14:34,226 [application.on_app_window_destroyed:346]:Removed window InstallerWindowmeteor-butterfly-and-a-blade
INFO     2024-08-19 08:14:50,259 [application.do_shutdown:898]:Shutting down Lutris

System Information

[System]
OS:              Debian GNU/Linux 12 bookworm
Arch:            x86_64
Kernel:          6.1.0-23-amd64
Desktop:         GNOME
Display Server:  x11

[CPU]
Vendor:          GenuineIntel
Model:           12th Gen Intel(R) Core(TM) i7-12700H
Physical cores:  14
Logical cores:   20

[Memory]
RAM:             15.3 GB
Swap:            1.9 GB

[Graphics]
Vendor:          Intel
OpenGL Renderer: Mesa Intel(R) Graphics (ADL GT2)
OpenGL Version:  4.6 (Compatibility Profile) Mesa 22.3.6
OpenGL Core:     4.6 (Core Profile) Mesa 22.3.6
OpenGL ES:       OpenGL ES 3.2 Mesa 22.3.6
Vulkan:          Supported

Media (optional)

No response

danieljohnson2 commented 1 month ago

I think what we're seeing is that your filesystem is not using UTF-8, and the problem is that there's really no way to know what the encoding is- you could even use different encoding for different filenames. Say, you could encode filenames according to your current locale, and then change that weekly.

That's really completely untenable. The only practical answer is to use UTF-8 for every locale on Linux.

What we can actually do for you is to ignore the non-ASCII bits when this happens, since what we are looking for is all ASCII anyway. Most legacy encodings we could see are supersets of ASCII, so the ASCII bits are still readable.

Please download this branch here which tries to do this. Just run the bin/lutris script within- no need to install- and see if that helps.

I'll merge this if it helps. But if I am right, this is a band-aid on a bullet would. You really need to move to UTF-8.

shiluotang commented 1 month ago

I think what we're seeing is that your filesystem is not using UTF-8, and the problem is that there's really no way to know what the encoding is- you could even use different encoding for different filenames. Say, you could encode filenames according to your current locale, and then change that weekly.

That's really completely untenable. The only practical answer is to use UTF-8 for every locale on Linux.

What we can actually do for you is to ignore the non-ASCII bits when this happens, since what we are looking for is all ASCII anyway. Most legacy encodings we could see are supersets of ASCII, so the ASCII bits are still readable.

Please download this branch here which tries to do this. Just run the bin/lutris script within- no need to install- and see if that helps.

I'll merge this if it helps. But if I am right, this is a band-aid on a bullet would. You really need to move to UTF-8.

  1. my debian 12 is already setup with UTF-8, and locale command gives following output 
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

So how could it be more UTF-8?

  1. as it's mentioned above, the /proc/{pid}/cmdline file has no encoding problem on my computer, you may give it a try
  2. I may try the fix when network condition better
danieljohnson2 commented 1 month ago

I don't know how I can try your computer's /proc directory; the error is showing something that's not UTF-8 in there. I am only guess about what it is.

I hope my branch will allow Lutris to just skip over that, whatever it is. Please try it when you can.

shiluotang commented 1 month ago

I don't know how I can try your computer's /proc directory; the error is showing something that's not UTF-8 in there. I am only guess about what it is.

I hope my branch will allow Lutris to just skip over that, whatever it is. Please try it when you can.

  1. the /proc/{pid}/cmdline file exists on many linux distributions
  2. i'm not asking you to try this file on my computer.
  3. get a debian 12 installation or some other linux distribution setup with utf-8 and write a simple long live program with non ascii filename, run it. then it can be verified.
shiluotang commented 1 month ago
  1. this branch has been tried, and it works without previous error.
  2. changes of this branch is just replace non decodeable bytes which is exactly what "skip over" means
  3. i've not read source code thoroughly. if filename is concerned, /proc/{pid}/cmdline file is still a good option
danieljohnson2 commented 1 month ago

OK, I've merged it. 0.5.18 should contain the fix, and I believe it will work no matter what the source of the mysterious non-UTF-8 command line data is.