search

ValveSoftware / steam-runtime

A runtime environment for Steam applications
Other
1.18k stars 86 forks source link

gconv modules fail to load from /usr/lib64/gconv on Void Linux + SLR #680

Closed oreo639 closed 2 weeks ago

oreo639 commented 1 month ago

Your system information

Please describe your issue in as much detail as possible:

Attempting to run 64-bit software that relies on iconv(), including Easy Anticheat, causes iconv() to fail.

On Void linux the $(libdir) used when compiling glibc is /usr/lib{bits} which results in glibc attempting to search for modules in /usr/lib64/gconv, however, pressure-vessel only mounts the gconv modules to /usr/lib/gconv and /usr/lib32/gconv leaving /usr/lib64/gconv as an empty directory causing iconv() to fail to find any gconv modules.

Steps for reproducing this issue:

  1. Compile glibc with --libdir=/usr/lib{bits}: https://github.com/void-linux/void-packages/blob/master/srcpkgs/glibc/template#L67
  2. Install steam and the steam linux runtime
  3. Download the following example and compile with gcc iconv.c -o iconv: https://gist.github.com/lesstif/626451b603e7e8a8d6b65df8eced29f9
  4. Run "$HOME/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier"/_v2-entry-point --verb=waitforexitandrun -- ./iconv and note that it outputs "not supported code"
  5. Note that the above command works if you add GCONV_PATH="/usr/lib/gconv" to the beginning.

See also: https://github.com/void-linux/void-packages/issues/41388#issuecomment-2227122490

smcv commented 1 month ago

To confirm some facts about how your system is working, please capture a verbose log from a Void Linux system. The usual way to do this is to set the launch options of a game that uses SLR or Proton to STEAM_LINUX_RUNTIME_VERBOSE=1 STEAM_LINUX_RUNTIME_LOG=1 %command%, then launch that game, then look for .../SteamLinuxRuntime_soldier/var/slr-latest.log or .../SteamLinuxRuntime_sniper/var/slr-latest.log depending on which SLR/Proton version the game uses.

Or if you're more comfortable with invoking SLR more directly, you can use a command like:

STEAM_LINUX_RUNTIME_VERBOSE=1 \
$HOME/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier'/_v2-entry-point \
--verb=waitforexitandrun -- ./iconv

and redirect its output to a log file.

smcv commented 1 month ago

The Steam Linux Runtime framework currently supports several models for how the multiarch/multilib ${libdir} could be arranged (in each case below I've mentioned 32- and 64-bit in that order):

... and then assumes that the directory for gconv modules is ${libdir}/gconv in all cases.

It looks as though Void Linux is using a mixture of the Arch and ClearLinux layouts: the actual shared libraries are like Arch (32-bit ${libdir} is /usr/lib32, 64-bit ${libdir} is /usr/lib), but gconv modules are like ClearLinux (they're loaded from /usr/lib32 or /usr/lib64, and never from /usr/lib). Is that accurate?

okawo80085 commented 1 month ago

It looks as though Void Linux is using a mixture of the Arch and ClearLinux layouts: the actual shared libraries are like Arch (32-bit ${libdir} is /usr/lib32, 64-bit ${libdir} is /usr/lib), but gconv modules are like ClearLinux (they're loaded from /usr/lib32 or /usr/lib64, and never from /usr/lib). Is that accurate?

Not exactly, on Void Linux all of the above are present

$ ls -l /usr
...
drwxr-xr-x 168 root root 176128 Jul 12 21:00 lib
drwxr-xr-x  15 root root  24576 Jul  6 23:58 lib32
lrwxrwxrwx   1 root root      3 Feb 12 19:13 lib64 -> lib
...

However /usr/lib64 is a symlink to /usr/lib, /usr/lib32/gconv is present as well

If its recognizing the system as ClearLinux, is it failing to follow a symlink? If its recognizing the system as Arch what is it missing?

okawo80085 commented 1 month ago

To confirm some facts about how your system is working, please capture a verbose log from a Void Linux system. The usual way to do this is to set the launch options of a game that uses SLR or Proton to STEAM_LINUX_RUNTIME_VERBOSE=1 STEAM_LINUX_RUNTIME_LOG=1 %command%, then launch that game, then look for .../SteamLinuxRuntime_soldier/var/slr-latest.log or .../SteamLinuxRuntime_sniper/var/slr-latest.log depending on which SLR/Proton version the game uses.

Or if you're more comfortable with invoking SLR more directly, you can use a command like:

STEAM_LINUX_RUNTIME_VERBOSE=1 \
$HOME/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier'/_v2-entry-point \
--verb=waitforexitandrun -- ./iconv

and redirect its output to a log file.

Will do later

smcv commented 1 month ago

pressure-vessel doesn't really specifically recognize the system as anything (that would scale really poorly - we'd have to implement specific code for everyone's favourite distro, and there are hundreds of major and minor variants). Instead, it assumes that the realpath() of a library reflects the canonical path that the distro prefers to use.

This is usually true on most distros - for instance on Arch Linux (which has the same on-disk physical layout, with a symlink lib64 -> lib), the canonical path that is hard-coded into libraries that want to load plugins is something like /usr/lib/gconv.

The difference here is that on Void Linux it seems like the realpath() of glibc will be /usr/lib/libc.so.6 or similar (just like Arch), but unlike Arch, glibc wants to load its plugins from the hard-coded path /usr/lib64/gconv which doesn't match that.

I think the shortest path to fixing this is likely to be teaching pressure-vessel to set GCONV_PATH itself, internally; so that wherever your gconv modules turn up inside the container, glibc will be forced to load them.

if you add GCONV_PATH="/usr/lib/gconv" to the beginning

I think a better workaround would be GCONV_PATH=/usr/lib/gconv:/usr/lib32/gconv, which would also work for 32-bit games.

If you set the Launch Options of a game that uses EAC to

GCONV_PATH=/usr/lib/gconv:/usr/lib32/gconv %command%

does that work around the EAC issue? If yes, then that would be the short term workaround that I would suggest.

smcv commented 1 month ago

@kisak-valve or @oreo639, please could you retitle this to "gconv modules fail to load from /usr/lib64/gconv on Void Linux + SLR" to set its scope?

This looks like a distro-specific issue affecting Void Linux only (and maybe a few other distros that are unusual in the same way), and the suggested workaround is hopefully correct for Void Linux but would not work for other major distro families. I don't want to get non-Void-Linux users replying to this issue with unrelated EAC problems, that would just cause confusion and make it take longer to fix anything!

smcv commented 1 month ago

If Void Linux developers want to make it not be an outlier in this way, the way to achieve that would be to ensure that the path you hard-code into your glibc as the place it wants to look for its gconv modules is the one that doesn't involve traversing a symlink: /usr/lib32/gconv for 32-bit (same as now), but /usr/lib/gconv for 64-bit (replacing the current /usr/lib64/gconv). That would result in this particular part of Void Linux being "the same shape" as e.g. Arch, which is something that already works.

I don't know whether that's an option or whether it would break some other aspect of the OS design (this is the first time I've really looked at Void Linux!) but it's a simplification that might be helpful in general.

As a side benefit, that would also make loading the modules very very slightly more efficient (although likely not enough to show up in anything except a highly artificial benchmark).

oreo639 commented 1 month ago

I don't know whether that's an option or whether it would break some other aspect of the OS design (this is the first time I've really looked at Void Linux!) but it's a simplification that might be helpful in general.

Not really feasible afaict. Arch handles it by just having lib32-* packages specifically for 64-bit multiarch. Our -32bit packages are generated while building packages for the i686 target so if we made /usr/lib for glibc i686, then that would break glibc-32bit (since /usr/lib is 64-bit on multiarch), currently we specify /usr/lib32 but on i686, that is a symlink, for example.

The other option to fix it on our end would be to make /usr/lib32 and /usr/lib64 not symlinks and make /usr/lib a symlink, although that would require rebuilding everything.

smcv commented 1 month ago

Our -32bit packages are generated while building packages for the i686 target so if we made /usr/lib for glibc i686, then that would break glibc-32bit (since /usr/lib is 64-bit on multiarch), currently we specify /usr/lib32 but on i686, that is a symlink, for example.

I see the problem. You want the architectures to be "symmetrical", so that you can use literally the same binaries for a purely i686 system that are also used as the 32-bit compatibility layer on an x86_64 system (like e.g. Debian does); but you also want the directory that physically contains the libraries for the system's primary architecture to be /usr/lib (like e.g. Arch does). I don't see a way to reconcile those two demands without sometimes having to load modules from a path that is a symlink, but that means SLR can't automatically understand which of the paths you consider to be the canonical one.

The other option to fix it on our end would be to make /usr/lib32 and /usr/lib64 not symlinks and make /usr/lib a symlink, although that would require rebuilding everything.

In our taxonomy of "which OS has which quirks?", that would result in the OS being "the same shape" as ClearLinux and Solus, rather than Arch; but I realise the need for a mass rebuild probably makes it unachievable.

smcv commented 1 month ago

On an affected system, with any GCONV_PATH workarounds removed, please try replacing SteamLinuxRuntime_sniper/pressure-vessel/ and/or SteamLinuxRuntime_soldier/pressure-vessel/ (whichever runtime is in use) with the result of unpacking this: https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/jobs/618685/artifacts/raw/_build/pressure-vessel-bin.tar.gz

For Proton/EAC users, it's likely to be SteamLinuxRuntime_sniper/pressure-vessel/ that is the important one (used by Proton 8.0 and up). For manual testing, hopefully you know which one you're using. If in doubt, replace both.

To return to an official build of pressure-vessel, use Verify Integrity on the "Steam Linux Runtime 3.0 (sniper)" and "Steam Linux Runtime 2.0 (soldier)" tools in your Steam library.

(This ends up setting GCONV_PATH=/usr/lib/gconv:/usr/lib32/gconv inside the container, but does it automatically. Test-build taken from !726 v2.)

oreo639 commented 1 month ago

Unfortunately, it seems that when passing GCONV_PATH=/usr/lib/gconv:/usr/lib32/gconv to a 32-bit application, it tries to open /usr/lib/gconv modules, fails, and then gives up. The same happens when passing /usr/lib32/gconv to a 64-bit application.

glibc reads the configuration files for all the specified gconv paths (with the default ones last), however it seems to always try to open the module from the first path possible and simply fail if that module doesn't load.

smcv commented 1 month ago

Ugh, that's really annoying, and seems like an oversight in glibc.

As a plan B, it might be possible to make /usr/lib64/gconv a symlink to /usr/lib/gconv, which I think would make Void Linux's glibc happy?

okawo80085 commented 1 month ago

/usr/lib64/gconv is already a symlink to /usr/lib/gconv on void, doesn't seem to make it more happy :/

smcv commented 1 month ago

That's the situation on the host system, but I'm talking about the situation that the container runtime framework sets up inside the container (which is done programmatically). I'm looking into whether it's feasible to achieve that.

smcv commented 1 month ago

While I'm looking into that, the information I asked for in https://github.com/ValveSoftware/steam-runtime/issues/680#issuecomment-2228252084 would still be a useful thing for someone to capture, so we have a better understanding of Void Linux and how it interacts with the container runtime framework.

caszuu commented 1 month ago

Here's the gist of invoking iconv binary on my machine

okawo80085 commented 1 month ago

While I'm looking into that, the information I asked for in #680 (comment) would still be a useful thing for someone to capture, so we have a better understanding of Void Linux and how it interacts with the container runtime framework.

I'll do that as soon as i can, but no eta cuz we are getting unstable blackouts here

smcv commented 1 month ago

No need now, @caszuu provided equivalent information (I'm assuming their system is the same as yours in all the ways that matter here).

smcv commented 1 month ago

Here's a new build to try: https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/jobs/620187/artifacts/raw/_build/pressure-vessel-bin.tar.gz (from !726 v3). Same test procedure as described in https://github.com/ValveSoftware/steam-runtime/issues/680#issuecomment-2233774154.

Whether this works or not, a new log as per https://github.com/ValveSoftware/steam-runtime/issues/680#issuecomment-2228252084 with the version under test, similar to the one @caszuu provided in https://github.com/ValveSoftware/steam-runtime/issues/680#issuecomment-2236522395, would be useful information.

If all goes well, on Void Linux this should end up putting symlinks to the host system's /usr/lib/gconv at both /usr/lib/gconv and /usr/lib64/gconv inside the container, ensuring that whichever of those paths Void Linux's glibc wants to use, it works. The log should say something like:

pressure-vessel-wrap[...]: D: Checking for gconv in /usr/lib64/gconv
...
pressure-vessel-wrap[...]: D: Checking for gconv in /usr/lib/gconv
...
pressure-vessel-wrap[...]: D: Checking for gconv in /usr/lib32/gconv
...
pressure-vessel-wrap[...]: D: Making provider gconv modules visible in container
pressure-vessel-wrap[...]: D: Removing "${container}//usr/lib/gconv"
pressure-vessel-wrap[...]: D: Creating symlink "${container}//usr/lib/gconv" -> "/run/host/usr/lib/gconv"
pressure-vessel-wrap[...]: D: Removing "${container}//usr/lib32/gconv"
pressure-vessel-wrap[...]: D: Creating symlink "${container}//usr/lib32/gconv" -> "/run/host/usr/lib32/gconv"
pressure-vessel-wrap[...]: D: Removing "${container}//usr/lib64/gconv"
pressure-vessel-wrap[...]: D: Creating symlink "${container}//usr/lib64/gconv" -> "/run/host/usr/lib64/gconv"
caszuu commented 1 month ago

Can confirm that the iconv binary (both 64 and 32 bit) and EAC games work as expected when using the patch!

@oreo639 Also unrelated to Steam, the glibc-32bit package seems be missing a gconv-modules-extra.conf without which the iconv binary fails to run. Should i make an issue about this?

oreo639 commented 1 month ago

Whether this works or not, a new log as per #680 (comment) with the version under test, similar to the one @caszuu provided in #680 (comment), would be useful information.

Seems to work: https://gist.github.com/oreo639/26fd5da965b64c4d01c6a40d8b4011c7

Also unrelated to Steam, the glibc-32bit package seems be missing a gconv-modules-extra.conf without which the iconv binary fails to run. Should i make an issue about this?

Thanks for bringing that up. I was confused how I had it but I realize what happened. Should be resolved: https://github.com/void-linux/void-packages/commit/4c973e67f4ae521b6e8a5d2cc7c15df71e95bf22

smcv commented 1 month ago

Thanks, a change equivalent to the test-build above is queued for inclusion in a future release. I'll update this issue when that becomes available to the public.

Until that ships, the workaround is to use launch options GCONV_PATH=/usr/lib/gconv %command% on Void Linux for 64-bit games that use EAC or iconv().

smcv commented 3 weeks ago

Thanks, a change equivalent to the test-build above is queued for inclusion in a future release. I'll update this issue when that becomes available to the public.

This shipped as a beta yesterday. Please try one of the affected games with the client_beta branch of the "Steam Linux Runtime 3.0 (sniper)" compatibility tool, removing the GCONV_PATH=/usr/lib/gconv %command% workaround.

Using the beta version of SLR is the same as using the beta branch of a game, but instead of looking for the game in your Steam library, you should look for "Steam Linux Runtime 3.0 (sniper)".

The updated version will be copied to the default branch at some point in the future, after it has had enough testing (usually 2-4 weeks, but it depends what else is happening, and I cannot predict a specific date).

Because Void Linux is a relatively unusual Linux distribution, it would be useful if some Void Linux users or developers could continue to use the Steam client beta and the SLR client_beta branch, so that you have some early warning of future changes. If a change in a beta makes something stop working, please report it as an issue, then try switching back to the default (non-beta) branch - knowing whether that's a successful workaround is a very useful piece of information when we are trying to solve a problem.

oreo639 commented 3 weeks ago

Thanks, the the latest SRL beta does appear to work.

smcv commented 2 weeks ago

The updated version will be copied to the default branch at some point in the future, after it has had enough testing

This has now happened, so I think this issue can be closed (but, like I said, it would be helpful if some Void Linux users or developers could continue to use the beta so we have early warning of any Void-specific issues).