search

FEX-Emu / FEX

A fast usermode x86 and x86-64 emulator for Arm64 Linux
https://fex-emu.com
MIT License
2.31k stars 123 forks source link

Change the LD_LIBRARY_PATH for native libraries? #3324

Closed MastaG closed 10 months ago

MastaG commented 11 months ago

Hi there,

I've just installed FEX-emu on my Android 12 phone running Ubuntu 23.10 in a Proot environment. In order to get proper OpenGL (using zink) and Vulkan using freedreno's turnip, I've manually compiled mesa from source and installed it in /usr/local/lib64 and /usr/local/lib.

So I've set:

LIBGL_DRIVERS_PATH=/usr/local/lib64/dri:/usr/local/lib/dri
VK_ICD_FILENAMES=/usr/local/share/vulkan/icd.d/freedreno_icd.aarch64.json:/usr/local/share/vulkan/icd.d/freedreno_icd.armv7l.json

And I've also created a /etc/ld.conf.so.d/000-local.conf with:

/usr/local/lib64
/usr/local/lib

This works great for native things like glxinfo, vkcube and supertuxkart:

$ MESA_LOADER_DRIVER_OVERRIDE=zink glxinfo
...
OpenGL vendor string: Mesa
OpenGL renderer string: zink Vulkan 1.3(Turnip Adreno (TM) 640 (MESA_TURNIP))
OpenGL core profile version string: 4.6 (Core Profile) Mesa 24.0.0-devel (git-a921a69010)
OpenGL core profile shading language version string: 4.60
OpenGL core profile context flags: no-error
OpenGL core profile profile mask: core profile
...
$ vkcube
Selected GPU 0: Turnip Adreno (TM) 640, type: IntegratedGpu

However, when I run steam, it doesn't seem to load the native libGL from /usr/local/lib{,64}:

$ MESA_LOADER_DRIVER_OVERRIDE=zink FEXBash steam

steam.sh[8786]: Running Steam on ubuntu 23.10 64-bit
steam.sh[8786]: STEAM_RUNTIME is enabled by the user
setup.sh[8858]: Steam runtime environment up-to-date!
steam.sh[8786]: Steam client's requirements are satisfied
[2023-12-12 14:00:46] Startup - updater built Dec  8 2023 00:32:59
[2023-12-12 14:00:46] Startup - Steam Client launched with: '/home/mastag/.local/share/Steam/ubuntu12_32/steam'
12/12 14:00:46 Init: Installing breakpad exception handler for appid(steam)/version(1702079146)/tid(8884)
MESA-LOADER: failed to open zink: /usr/local/lib/dri/zink_dri.so: cannot open shared object file: No such file or directory (search paths /usr/local/lib64/dri:/usr/local/lib/dri, suffix _dri)
failed to load driver: zink
Looks like steam didn't shutdown cleanly, scheduling immediate update check
[2023-12-12 14:00:53] Loading cached metrics from disk (/home/mastag/.local/share/Steam/package/steam_client_metrics.bin)
[2023-12-12 14:00:53] Failed to load cached hosts file (File 'update_hosts_cached.vdf' not found), using defaults
[2023-12-12 14:00:53] Using the following download hosts for Public, Realm steamglobal
[2023-12-12 14:00:53] 1. https://cdn.steamstatic.com, /client/, Realm 'steamglobal', weight was 1, source = 'baked in'
[2023-12-12 14:00:53] Checking for update on startup
[2023-12-12 14:00:53] Checking for available updates...
[2023-12-12 14:00:53] Downloading manifest: https://cdn.steamstatic.com/client/steam_client_ubuntu12
[2023-12-12 14:00:53] Manifest download: send request
[2023-12-12 14:00:53] Manifest download: waiting for download to finish
[2023-12-12 14:00:54] Manifest download: finished
[2023-12-12 14:00:54] Download skipped: /client/steam_client_ubuntu12 version 1702079146, installed version 1702079146, existing pending version 0
[2023-12-12 14:00:54] Nothing to do
[2023-12-12 14:00:54] Verifying installation...
[2023-12-12 14:00:54] Performing checksum verification of executable files
[2023-12-12 14:01:08] Verification complete

Steam logging initialized: directory: /home/mastag/.local/share/Steam/logs

XRRGetOutputInfo Workaround: initialized with override: 0 real: 0xf5f3a9c0
XRRGetCrtcInfo Workaround: initialized with override: 0 real: 0xf5f391f0
lspci: Cannot open /proc/bus/pci/devices
MESA-LOADER: failed to open zink: /usr/local/lib/dri/zink_dri.so: cannot open shared object file: No such file or directory (search paths /usr/local/lib64/dri:/usr/local/lib/dri, suffix _dri)
failed to load driver: zink
lspci: Cannot open /proc/bus/pci/devices
MESA-LOADER: failed to open zink: /usr/local/lib/dri/zink_dri.so: cannot open shared object file: No such file or directory (search paths /usr/local/lib64/dri:/usr/local/lib/dri, suffix _dri)
failed to load driver: zink
steamwebhelper.sh[8923]: Runtime for steamwebhelper: defaulting to /home/mastag/.local/share/Steam/ubuntu12_64/steam-runtime-heavy
steamwebhelper.sh[8923]: glibc >= 2.34, partially disabling sandbox until CEF supports clone3()
MESA-LOADER: failed to open zink: /usr/local/lib/dri/zink_dri.so: cannot open shared object file: No such file or directory (search paths /usr/local/lib64/dri:/usr/local/lib/dri, suffix _dri)
failed to load driver: zink
glXChooseVisual failed
glXChooseVisual failedsrc/steamUI/spewmanager.cpp (184) : Assertion Failed: Error: glXChooseVisual failed
src/steamUI/spewmanager.cpp (184) : Assertion Failed: Error: glXChooseVisual failed
Error:  glXChooseVisual failed12/12 14:01:12 Failed writing minidump, nothing to upload.
/home/mastag/.local/share/Steam/steam.sh: line 798:  8884 Segmentation fault      "$STEAMROOT/$STEAMEXEPATH" "$@"

Any ideas?

Sonicadvance1 commented 11 months ago

Sounds like you need to install the x86 and x86-64 custom mesa libraries in to the Ubuntu rootfs paths as well. Since it is trying to search for zink in $ROOTFS/usr/local/lib/dri/zink_dri.so which doesn't exist. Since the mesa libraries we have installed in the rootfs are under $ROOTFS/usr/lib/dri/zink_dri.so instead.

MastaG commented 11 months ago

Thanks @Sonicadvance1 But do they need to be the exact same libraries? Or can I just copy the stock ones from /usr/lib to /use/local/lib? Because in the first case I'll have to compile the exact same source on a x86-64 host running Ubuntu 23.10 first. The ones from the image are probably something like mesa 23.2 and the custom mesa is 24.0.0-devel.

Sonicadvance1 commented 11 months ago

I believe mesa hardcodes some paths so just moving them from /usr/ to /usr/local/ might cause issues. Could give it a try though.

The Mesa version we are currently shipping in our rootfs images are Mesa 23.3.0, we overwrite the version that Ubuntu ships.

MastaG commented 11 months ago

You're right, so here's what I did. First I setup a Ubuntu 23.10 x86-64 VM on my laptop and I've rsynced all of the modified mesa sources to it. Then I configured and compiled mesa with the exact same options for both 32bit and 64bit. Finally I rsynced the /usr/local folder from my VM into the FEX' RootFS folder.

So I now have:

$ file ~/.fex-emu/RootFS/Ubuntu_23_10/usr/local/lib64/libGL.so.1.2.0 
/home/mastag/.fex-emu/RootFS/Ubuntu_23_10/usr/local/lib64/libGL.so.1.2.0: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=c7a79964ae9f192d9b7a193c081898bfb4da5803, not stripped

$ file ~/.fex-emu/RootFS/Ubuntu_23_10/usr/local/lib/libGL.so.1.2.0 
/home/mastag/.fex-emu/RootFS/Ubuntu_23_10/usr/local/lib/libGL.so.1.2.0: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, BuildID[sha1]=a2069cd08f182ee6e2ce06cf6b3a71841ebfaab0, not stripped

They also seem to be working from within FEXBash:

$ FEXBash 
FEXBash-mastag@localhost:~> LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib MESA_LOADER_DRIVER_OVERRIDE=zink glxinfo
name of display: :0
ATTENTION: default value of option vblank_mode overridden by environment.
display: :0  screen: 0
direct rendering: Yes
server glx vendor string: SGI
server glx version string: 1.4
server glx extensions:
    GLX_ARB_create_context, GLX_ARB_create_context_no_error, 
    GLX_ARB_create_context_profile, GLX_ARB_fbconfig_float, 
    GLX_ARB_framebuffer_sRGB, GLX_ARB_multisample, 
    GLX_EXT_create_context_es2_profile, GLX_EXT_create_context_es_profile, 
    GLX_EXT_fbconfig_packed_float, GLX_EXT_framebuffer_sRGB, 
    GLX_EXT_get_drawable_type, GLX_EXT_libglvnd, GLX_EXT_no_config_context, 
    GLX_EXT_texture_from_pixmap, GLX_EXT_visual_info, GLX_EXT_visual_rating, 
    GLX_MESA_copy_sub_buffer, GLX_OML_swap_method, GLX_SGIS_multisample, 
    GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, GLX_SGIX_visual_select_group, 
    GLX_SGI_make_current_read
client glx vendor string: Mesa Project and SGI
client glx version string: 1.4
client glx extensions:
    GLX_ARB_context_flush_control, GLX_ARB_create_context, 
    GLX_ARB_create_context_no_error, GLX_ARB_create_context_profile, 
    GLX_ARB_create_context_robustness, GLX_ARB_fbconfig_float, 
    GLX_ARB_framebuffer_sRGB, GLX_ARB_get_proc_address, GLX_ARB_multisample, 
    GLX_ATI_pixel_format_float, GLX_EXT_buffer_age, 
    GLX_EXT_create_context_es2_profile, GLX_EXT_create_context_es_profile, 
    GLX_EXT_fbconfig_packed_float, GLX_EXT_framebuffer_sRGB, 
    GLX_EXT_import_context, GLX_EXT_no_config_context, GLX_EXT_swap_control, 
    GLX_EXT_swap_control_tear, GLX_EXT_texture_from_pixmap, 
    GLX_EXT_visual_info, GLX_EXT_visual_rating, GLX_INTEL_swap_event, 
    GLX_MESA_copy_sub_buffer, GLX_MESA_gl_interop, GLX_MESA_query_renderer, 
    GLX_MESA_swap_control, GLX_NV_float_buffer, GLX_OML_sync_control, 
    GLX_SGIS_multisample, GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, 
    GLX_SGIX_visual_select_group, GLX_SGI_make_current_read, 
    GLX_SGI_swap_control, GLX_SGI_video_sync
GLX version: 1.4
GLX extensions:
    GLX_ARB_create_context, GLX_ARB_create_context_no_error, 
    GLX_ARB_create_context_profile, GLX_ARB_fbconfig_float, 
    GLX_ARB_framebuffer_sRGB, GLX_ARB_get_proc_address, GLX_ARB_multisample, 
    GLX_EXT_buffer_age, GLX_EXT_create_context_es2_profile, 
    GLX_EXT_create_context_es_profile, GLX_EXT_fbconfig_packed_float, 
    GLX_EXT_framebuffer_sRGB, GLX_EXT_no_config_context, GLX_EXT_swap_control, 
    GLX_EXT_texture_from_pixmap, GLX_EXT_visual_info, GLX_EXT_visual_rating, 
    GLX_MESA_query_renderer, GLX_MESA_swap_control, GLX_SGIS_multisample, 
    GLX_SGIX_fbconfig, GLX_SGIX_pbuffer, GLX_SGIX_visual_select_group, 
    GLX_SGI_make_current_read
Extended renderer info (GLX_MESA_query_renderer):
    Vendor: Mesa (0x5143)
    Device: zink Vulkan 1.3(Turnip Adreno (TM) 640 (MESA_TURNIP)) (0x6040001)
    Version: 24.0.0
    Accelerated: yes
    Video memory: 8585MB
    Unified memory: yes
    Preferred profile: core (0x1)
    Max core profile version: 4.6
    Max compat profile version: 4.6
    Max GLES1 profile version: 1.1
    Max GLES[23] profile version: 3.2
Memory info (GL_ATI_meminfo):
    VBO free memory - total: 8519 MB, largest block: 8519 MB
    VBO free aux. memory - total: 0 MB, largest block: 0 MB
    Texture free memory - total: 8519 MB, largest block: 8519 MB
    Texture free aux. memory - total: 0 MB, largest block: 0 MB
    Renderbuffer free memory - total: 8519 MB, largest block: 8519 MB
    Renderbuffer free aux. memory - total: 0 MB, largest block: 0 MB
Memory info (GL_NVX_gpu_memory_info):
    Dedicated video memory: 8585 MB
    Total available memory: 8585 MB
    Currently available dedicated video memory: 8519 MB
OpenGL vendor string: Mesa
OpenGL renderer string: zink Vulkan 1.3(Turnip Adreno (TM) 640 (MESA_TURNIP))
OpenGL core profile version string: 4.6 (Core Profile) Mesa 24.0.0-devel
OpenGL core profile shading language version string: 4.60
OpenGL core profile context flags: no-error
OpenGL core profile profile mask: core profile
...

glxgears is also working from withtin FEXBash :) So far so good!

So I've created a small script to run steam:

$ cat /usr/local/bin/run_steam 
#!/bin/bash
export STEAMOS=1
export STEAM_RUNTIME=1
export DBUS_FATAL_WARNINGS=0
export MESA_LOADER_DRIVER_OVERRIDE=zink
export LD_LIBRARY_PATH=/usr/local/lib64:/usr/local/lib
exec taskset -c 4-7 ~/steam/bin/steam "$@"

However when running it, I eventually hit an illegal instruction:

$ FEXBash run_steam
bin_steam.sh[10373]: Setting up Steam content in /home/mastag/.local/share/Steam
steam.sh[10373]: Running Steam on ubuntu 23.10 64-bit
steam.sh[10373]: STEAM_RUNTIME is enabled by the user
setup.sh[10463]: Updating Steam runtime environment...

(process:10471): Gtk-WARNING **: 13:31:31.782: Locale not supported by C library.
    Using the fallback 'C' locale.

steam.sh[10373]: Can't find 'steam-runtime-check-requirements', continuing anyway
[2023-12-13 13:36:35] Startup - updater built Apr 28 2023 18:32:42
[2023-12-13 13:36:35] Startup - Steam Client launched with: '/home/mastag/.local/share/Steam/ubuntu12_32/steam'
ILocalize::AddFile() failed to load file "public/steambootstrapper_english.txt".
Installing breakpad exception handler for appid(steam)/version(0)/tid(15564)
No minidump written, nothing to upload.
/home/mastag/.local/share/Steam/steam.sh: line 798: 15564 Illegal instruction     "$STEAMROOT/$STEAMEXEPATH" "$@"

Any ideas?

Sonicadvance1 commented 11 months ago

Without some coredump it will hard to debug why it hit SIGILL. Could be something seccomp related. When running Steam on my Linux device I don't hit SIGILL with steam.

MastaG commented 11 months ago

Are there any instructions on how to create a coredump? normally I'd just run in through gdb and dump the backtrace.

Sonicadvance1 commented 11 months ago

Normally I would just use systemd-coredump

MastaG commented 10 months ago

Well I'm running FEX on termux-x11 using a Ubuntu 23.10 XFCE4 proot environment on a Android 12 device with a heavily outdated kernel (version 4.4.x). So things like systemd don't start. But the biggest setback is that it's not possible to setup proper binfmt support as this would require root access (breaking Google Play Integrity security on my device).

I'll close this for now as it's not possible to properly use coredump without systemd and also it seems the steam launch scripts will try to run the x86-64 binaries natively without resorting to FEX (or box64 for that matter) due to the lack of binfmt support. Altering the Steam runtime causes it to re-download everything (and they're a mess anyways.. way too over-engineered).

It just differs too much from a native linux distro.