steam: Proton 5.13 does not launch because of steam runtime

[nyanloutre]

Describe the bug Multiple errors when using SteamLinuxRuntime_soldier/run-in-soldier

This prevent the latest Proton (5.13) from running

To Reproduce The following commands are to run in a steam-run shell: steam-run /bin/bash

1. G_MESSAGES_DEBUG=all /mnt/hdd/Steam/steamapps/common/SteamLinuxRuntime_soldier/run-in-soldier -- echo coucou Error: x86_64-linux-gnu-capsule-capture-libs: code 2: open "/etc/ld.so.cache": No such file or directory
2. ldconfig -C /etc/ld.so.cache
3. G_MESSAGES_DEBUG=all /mnt/hdd/Steam/steamapps/common/SteamLinuxRuntime_soldier/run-in-soldier -- echo coucou Error: /run/pressure-vessel/pv-from-host/bin/pressure-vessel-adverb: error while loading shared libraries: libdl.so.2: cannot open shared object file: No such file or directory
4. G_MESSAGES_DEBUG=all /mnt/hdd/Steam/steamapps/common/SteamLinuxRuntime_soldier/run-in-soldier --filesystem=/nix -- echo coucou

It's finally printing

Expected behavior Should run natively from within steam and without manual tweaks

Additional context There is two problems to solve:

1. How to generate the /etc/ld.so.cache file
2. How to permanently add the --filesystem=/nix option to steam bwrap

Notify maintainers

Metadata Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

 - system: `"x86_64-linux"`
 - host os: `Linux 5.4.69, NixOS, 21.03.git.0da2a1a113e (Okapi)`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.3.7`
 - channels(root): `"nixos-21.03pre246543.24c9b05ac53"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
  - steam
# a list of nixos modules affected by the problem
module:

[nyanloutre]

Related issues:

• https://github.com/ValveSoftware/steam-runtime/issues/230
• https://github.com/NixOS/nixpkgs/issues/35387

[jhlkek]

Probably related, if I run Steam from a terminal window and try to run a game with Proton 5.13 I get this message before it (the game) crashes:

ERROR: ld.so: object '/home/user/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored. /home/user/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/pressure-vessel-adverb: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory ERROR: ld.so: object '/home/user/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored. /home/user/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/pressure-vessel-adverb: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

[smcv]

@nyanloutre, to clarify, if you apply these two workarounds (both of which are entirely unsupported and are done at your own risk, but they provide useful information):

• generate /etc/ld.so.cache
• hack the _start-container-in-background script to add --filesystem=/nix next to the other --filesystem options

is that sufficient to make Proton 5.13 work? (I suspect the answer is no, but I hope I'm wrong.)

How to permanently add the --filesystem=/nix option to steam bwrap

If it's that simple, then we can do the equivalent of that in the pressure-vessel source code. Making pressure-vessel work on non-FHS host distributions is not really on the radar right now (we have higher-priority issues to deal with, and fixes that apply to FHS distributions like Ubuntu benefit a lot more people than fixes for individual non-FHS distributions); but making a one-line change to share an extra filesystem has such a good effort/result and risk/result ratio that we can probably sneak it in to a future release anyway.

I'm not familiar with NixOS, except knowing that it's non-FHS. What's in /nix? Does it contain executables, libraries and static data belonging to OS-level packages, like /usr on a FHS system? Or does it contain system-specific variable data, like /etc, /var and /run on a FHS system? Or does it contain user data, like /home and /srv on a FHS system? Or some mixture of those?

If /nix only contains things that are reasonably similar to a FHS system's /usr, then we can certainly share it in the same situations where we'd share /usr.

If there are symbolic links into /nix from locations like /usr and /lib, are they absolute (/lib/foo -> /nix/path/to/foo), or relative (/lib/foo -> ../nix/path/to/foo), or a mixture? (This affects where we have to put it in the container to make it work: /nix, or /run/host/nix, or both.)

How to generate the /etc/ld.so.cache file

Where does a NixOS system normally keep its ld.so cache, and where will NixOS' glibc look for it?

Handling this "nicely" is likely to need C source code changes in libcapsule and pressure-vessel. See https://github.com/ValveSoftware/steam-runtime/issues/230#issuecomment-712133364 and https://gitlab.collabora.com/vivek/libcapsule/-/merge_requests/40 - the exact paths involved on Exherbo and NixOS are almost certainly different, but the general idea will be the same.

We do not have the resources to support non-FHS distros, but I'm hoping to get infrastructure in place in the C code so that distro developers can contribute a patch that adds their filesystems/ld.so.cache/etc. to a reasonably obvious list (probably with Exherbo as the initial example, just because someone has already told me how it works).

[nyanloutre]

Hello, and thanks for the help !

@nyanloutre, to clarify, if you apply these two workarounds (both of which are entirely unsupported and are done at your own risk, but they provide useful information):

* generate `/etc/ld.so.cache`

* hack the `_start-container-in-background` script to add `--filesystem=/nix` next to the other `--filesystem` options

is that sufficient to make Proton 5.13 work? (I suspect the answer is no, but I hope I'm wrong.)

It still doesn't launch any games, I have this error:

ERROR: ld.so: object '/home/paul/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
/mnt/hdd/Steam/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/pressure-vessel-adverb: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory
ERROR: ld.so: object '/home/paul/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.
/mnt/hdd/Steam/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/pressure-vessel-adverb: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

I'm not familiar with NixOS, except knowing that it's non-FHS. What's in /nix? Does it contain executables, libraries and static data belonging to OS-level packages, like /usr on a FHS system? Or does it contain system-specific variable data, like /etc, /var and /run on a FHS system? Or does it contain user data, like /home and /srv on a FHS system? Or some mixture of those?

In Nixos every binary and libs are located in /nix, everything else is only simlinks. To make Steam work, NixOS is using bwrap to give it an FHS environment. Unfortunatly it's not enough to make the steam runtime work.

If there are symbolic links into /nix from locations like /usr and /lib, are they absolute (/lib/foo -> /nix/path/to/foo), or relative (/lib/foo -> ../nix/path/to/foo), or a mixture? (This affects where we have to put it in the container to make it work: /nix, or /run/host/nix, or both.)

The links are always absolute and point to /nix/store/xxxxxxxxxxx-libfoo/...

Where does a NixOS system normally keep its ld.so cache, and where will NixOS' glibc look for it?

NixOS doesn't have an ld.so cache, but it may be possible to generate it in the wrapper script that already launch steam in a bwrap.

[nyanloutre]

I also managed to get this error by adding unset LD_PRELOAD in the _v2-entry-point script:

/usr/lib/x86_64-linux-gnu/gio/modules/libdconfsettings.so: undefined symbol: g_type_ensure
Failed to load module: /usr/lib/x86_64-linux-gnu/gio/modules/libdconfsettings.so

[smcv]

ERROR: ld.so: object '/home/paul/.local/share/Steam/ubuntu12_32/gameoverlayrenderer.so' from LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.

This is noise, you can safely ignore it. The way Steam sets up LD_PRELOAD is a bit weird, and will always cause these warnings.

/mnt/hdd/Steam/steamapps/common/SteamLinuxRuntime_soldier/pressure-vessel/bin/pressure-vessel-adverb: error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

This is the real problem. The container doesn't have the required ld.so, or ld.so.cache, or search paths, or something, to find the dependencies of your libstdc++.so.6.

(I'm not sure why pressure-vessel-adverb is loading C++ code - it should only depend on GLib and libelf, which are C.)

To make Steam work, NixOS is using bwrap to give it an FHS environment

That might not work particularly well with the container runtime, which also relies on bwrap.

What does that FHS environment look like? Is /usr populated with symlinks into /nix?

NixOS doesn't have an ld.so cache

That's unfortunate, because libcapsule relies on parsing the glibc ld.so cache to be able to find host system libraries. The alternative would be to hard-code a list of library directories and hope it matches the locations your glibc will look in.

it may be possible to generate it in the wrapper script that already launch steam in a bwrap

I think that would be a good idea.

/usr/lib/x86_64-linux-gnu/gio/modules/libdconfsettings.so: undefined symbol: g_type_ensure

You can safely ignore this. GLib automatically loads GIO modules, and finds the version on the host system, for which the copy of GLib provided with pressure-vessel (which is from Steam Runtime v1 for maximum portability) is too old; but it doesn't actually need any GIO modules, so it's harmless that they fail to load. We actually try to turn them off, but until recently the included copy of GLib was too old to have a good mechanism to do this. A future release will probably stop this happening, but it isn't a high priority.

adding unset LD_PRELOAD

This will break the Steam overlay, which used to be just the overlay used for achievement popups, the Shift+Escape screen, etc. but is gradually accumulating more important functionality (game controller remapping, streaming, probably more).

[L-as]

I made an FHS like so:

with import <nixpkgs> {};
buildEnv {
    name = "steam-env";
    paths = [ gcc-unwrapped.lib glib.out json-glib.out ];
}

and added the lib of this FHS to the rpath of pressure-vessel-adverb, which seems to have gotten it further, but adding all of the required libraries causes it to fail on libstdc++ again! I checked with strace, and it fails with this:

1226  openat(AT_FDCWD, "/nix/store/9df65igwjmf2wbw0gbrrgair6piqjgmi-glibc-2.31/lib/libstdc++.so.6", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

It opens libstdc++ from the rpath several times, but once tries to open it from here, but this output has no libstdc++, so it obviously fails. I am honestly not sure why it only tries this path, and I did try adding the FHS to LD_LIBRARY_PATH too to no effect.

[L-as]

@nyanloutre I can't reproduce your results with run-in-soldier. For me, the following command: steam-run sh -c ldconfig -C /etc/ld.so.cache -f /dev/null /lib; $HOME/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/run-in-soldier --filesystem=/nix -- echo test gives the following output:

/nix/store/x5jj29qcgyz4gn2ml076m7i305f4j31s-dconf-0.36.0-lib/lib/gio/modules/libdconfsettings.so: undefined symbol: g_type_ensure
Failed to load module: /nix/store/x5jj29qcgyz4gn2ml076m7i305f4j31s-dconf-0.36.0-lib/lib/gio/modules/libdconfsettings.so
pressure-vessel-wrap[4]: Using glibc from provider system for some but not all architectures! Arbitrarily using provider locales.
pressure-vessel-wrap[4]: Using libdrm.so.2 from provider system for some but not all architectures! Will take /usr/share/libdrm from provider.
/run/pressure-vessel/pv-from-host/bin/pressure-vessel-adverb: /../lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.30' not found (required by /nix/store/8nn8q8p6frw05q6d87kdig8x3c4pxly6-libselinux-2.9/lib/libselinux.so.1)

Wrt. to my above halfway workaround, even adding an ld.so.cache doesn't make it load from libstdc++ from another place. It always tries to load it from the same folder where the interpreter is located.

[L-as]

@smcv wrt. the FHS environment, it's actually /lib, /bin, etc. that are filled with absolute symlinks to /nix/..., and then /usr/x is just a symlink to /x for any x.

[smcv]

/nix/store/x5jj29qcgyz4gn2ml076m7i305f4j31s-dconf-0.36.0-lib/lib/gio/modules/libdconfsettings.so: undefined symbol: g_type_ensure

You can ignore that, it's harmless.

pressure-vessel-wrap[4]: Using glibc from provider system for some but not all architectures! Arbitrarily using provider locales. pressure-vessel-wrap[4]: Using libdrm.so.2 from provider system for some but not all architectures! Will take /usr/share/libdrm from provider.

Do you have both 32- and 64-bit graphics drivers on the host system? To use Steam, you really should (I'm surprised it works without involving pressure-vessel). The container has some basic Mesa graphics drivers (more as an accident of the dependency stack than anything deliberate), but they're from Debian 10, so they won't support very recent hardware.

/run/pressure-vessel/pv-from-host/bin/pressure-vessel-adverb: /../lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.30' not found (required by /nix/store/8nn8q8p6frw05q6d87kdig8x3c4pxly6-libselinux-2.9/lib/libselinux.so.1)

That's a new error, so we're making progress!

Please could you get a log with PRESSURE_VESSEL_VERBOSE=1 in the environment? That will make pressure-vessel give a lot more info about what it's thinking.

wrt. the FHS environment, it's actually /lib, /bin, etc. that are filled with absolute symlinks to /nix/..., and then /usr/x is just a symlink to /x for any x

That's the reverse of the /usr merge. pressure-vessel should cope equally well either way round, but if /usr was populated with a directory hierarchy (with symlinks to the libraries in /nix) and the compat symlinks were lib -> usr/lib, etc., that might get you onto more-commonly-tested code paths.

[moben]

For reference, here is a log with with PRESSURE_VESSEL_VERBOSE=1:

https://gist.github.com/moben/26dfe84ea4208bee42d98f40dd180c40

Full list of changes to get to this point:

1. Added /bin/ldconfig -C /etc/ld.so.cache to the top of SteamLinuxRuntime_soldier/_start-container-in-background.
2. Added --filesystem=/nix to the other --filesystem parameters in SteamLinuxRuntime_soldier/_start-container-in-background.
3. exported PRESSURE_VESSEL_VERBOSE=1 in SteamLinuxRuntime_soldier/_v2-entry-point as it wasn't propagated when set as a launch option in steam.

pressure-vessel-launcher[669]: Child 715 died: wait status 32512
pressure-vessel-launch[702]: child 715 exited: wait status 32512
pressure-vessel-launch[702]: child exit code 715: 127
pressure-vessel-launch[702]: Exiting with status 127
/data/src/clientdll/installscript_posix.cpp (419) : Assertion Failed: Standalone evaluator returned error code for app 418530
/data/src/clientdll/installscript_posix.cpp (419) : Assertion Failed: Standalone evaluator returned error code for app 418530

seems relevant. Looks like either libexpat is missing from the container that steam runs in on NixOS to simulate something close to an FHS environment or a python3 from the runtime is called but can't find the lib from the runtime?

Edit: Tried adding libexpat to the environment the steam client runs in, but the error remains. (might have made a mistake when adding the lib, https://github.com/moben/nixpkgs/commit/eada9fac16b09c7c45ea6994c206fd3f3203db33)

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/steam-games-unable-to-load-libstdc-so-6/9589/6

[moben]

Progress:

The error above is from running the actual proton script. As an test, I replaced it with a bash script, which results in bash failing to find libtinfo.so.6. The interesting part here is that the NixOS bash doesn't link against that but the one in soldier does.

So binaries are run from the runtime but the libraries aren't present.

Edit: /lib64/ld-linux-x86-64.so.2 is the one from the host. I think it's likely that this is due to the "reverse usrmerge", as @smcv suspected, as pressure-vessel mounts /usr but the binaries still hardcode /lib64. I think it makes sense to see if that can be switched in the NixOS wrapper for steam because it's unlikely that the binaries will change as the runtime is debian-based if I understand correctly.

[moben]

Same error when /lib, /bin, ... are symlinks to the directories in /usr.

Also tried with the client_beta branch for the soldier runtime as I'm running the beta of the steam client as well. (Same setup/workarounds otherwise). There we get these errors:

GameAction [AppID 418530, ActionID 2] : LaunchApp changed task to Completed with ""
Game removed: AppID 418530 "", ProcID 857 
Uploaded AppInterfaceStats to Steam
>>> Adding process 858 for game ID 418530
Exiting app 418530
No cached sticky mapping in ActivateActionSet.>>> Adding process 859 for game ID 418530
>>> Adding process 861 for game ID 418530
bwrap: Can't bind mount /oldroot/ on /newroot/: No such device
pressure-vessel-wrap[838]: Child process exited with code 1
pressure-vessel-wrap[838]: Exiting with status 1
ln: failed to create symbolic link '/tmp/SteamPVSockets.iS6Uav/SteamLinuxRuntime.4944e167c07d3386/socket' -> '': No such file or directory
pressure-vessel-adverb[829]: Child 831 exited with wait status 256
pressure-vessel-adverb[829]: Command exited with status 1
pressure-vessel-adverb[829]: No more child processes
pressure-vessel-adverb[829]: Exiting with status 1

Full log: https://gist.github.com/moben/aef8d7d553a25f352f77c27f2da4efba

[smcv]

Another new error, therefore more progress! :-)

bwrap: Can't bind mount /oldroot/ on /newroot/: No such device

I think this is probably pressure-vessel's bwrap getting confused by being run inside another bwrap. It uses pivot_root(), which has confusing and poorly-documented requirements.

[smcv]

For your reference, the way the runtime is meant to work is:

• Start from a Steam Runtime 2 'soldier' container. This is basically Debian 10 with a small number of patches, backports and additional packages.
• Look for graphics drivers (Mesa and/or NVIDIA-proprietary) in what we call the provider, which is normally meant to be the host system (but because you are running Steam in a bwrap container to get a FHS layout, what we thought was the host system is really your FHS container). Use them in the container. This works by mounting the provider on /run/host (or sometimes /run/gfx), and creating symbolic links in /overrides (or sometimes /usr/lib/pressure-vessel/overrides, see the source code) that point into that directory, and setting environment variables that make graphics driver loaders like GLVND and vulkan-loader look there in preference to /usr.
• Recursively look up the libraries that the graphics drivers depend on, such as glibc and libdrm.
• For each dependency library, look at the version in the provider and the version in the runtime, and try to figure out which one is newer. If the runtime version is newer, keep it. If the provider version is newer, drop it into /overrides in the same way we do for graphics drivers. We have to do this because libraries in the runtime are entitled to depend on libfoo >= the runtime version, and graphics drivers from the provider are entitled to depend on libfoo >= the provider version, so the only way we can guarantee to satisfy both of those dependencies is to choose whichever libfoo is newest.

This is all quite complicated, even on close-to-FHS systems like Fedora/SUSE (FHS, with 32-bit /usr/lib and 64-bit /usr/lib64), Debian/Ubuntu (FHS but with special multiarch library directories like /usr/lib/x86_64-linux-gnu), and Arch Linux (FHS but reversing the libQUAL directories so /usr/lib is 64-bit and /usr/lib32 is 32-bit). As you can imagine, when you run it on an entirely non-FHS layout (NixOS) and trick it into looking like FHS by using bwrap, everything gets even more confusing.

If the libraries we have stolen from your host system (graphics drivers and glibc) have any hard-coded paths in them, for example pointing into /nix, then we need to somehow make those paths exist inside the final container. This is part of why you have found that you need --filesystem=/nix.

We also need to make sure the glibc inside the final container can find standard Steam Runtime libraries like libtinfo.so.6 in their Debian-style locations, otherwise none of our executables will work. On close-to-FHS systems like Fedora, Debian and Arch, the host system's glibc will automatically search /etc/ld.so.cache, and our container ships with a /etc/ld.so.cache that knows about all the libraries we provide in the Steam Runtime, so it works. However, on more unusual systems like Exherbo where the cache filename hard-coded into ld.so is different, this won't work, and on entirely non-FHS systems like NixOS, all bets are off.

We have some work-in-progress to add a mode where instead of relying on LD_LIBRARY_PATH to load /overrides, we run ldconfig during container setup to regenerate ld.so.cache. This is primarily for the non-Proton use-case (native Linux games), to fix games that incorrectly clear the LD_LIBRARY_PATH, instead of correctly prepending or appending their bundled library directories to it. However, as a side-effect this will definitely help Exherbo (because we can make the new cache available at the filenames their glibc expects, in addition to the filename our glibc expected), and might also help you.

So, a question for you: is there a location where we could put a ld.so.cache, such that the ld.so from NixOS' glibc will automatically load it, and will find all the libraries listed there?

The other way we could potentially make this work is that at the moment, pressure-vessel cannot run successfully inside a Flatpak container, unless you disable all the sandboxing/security (which is a early prototype code rather than something we genuinely expect gamers to do). However, we want it to be able to run from the Flatpak version of Steam, either optionally or always using the freedesktop.org runtime (which is basically the same close-to-FHS layout as Debian) as the provider of graphics drivers. If NixOS has a working version of Flatpak, then this would provide a way to run Steam and pressure-vessel, bypassing all the issues that come with trying to reconcile NixOS' non-FHS layout with the close-to-FHS layout of the Steam Runtime.

[L-as]

As can be seen here, there is no fixed path from which ld.so.cache is read, it is instead read from /nix/store/<...>/etc/ld.so.cache. Even /etc/ld.so.preload is not used, and instead /etc/ld-nix.so.preload is read, as evident from here.

I am not sure why ld.so.cache is explicitly disallowed in this way in nixpkgs, though I assume that even if you removed that patch, nothing would break, since /etc/ld.so.cache doesn't exist anyway. I wonder if this is the only big thing that is preventing pressure-vessel from working.

[moben]

Actually there is no ld.so.cache as far as I can tell. All nix-built binaries have an RPATH that includes all their dependencies. This allows installing multiple versions of the dependencies on the system (but requires a lot of rebuilds).

This also means that normally there are no libraries in the FHS system libdirs, so it really is a rather peculiar setup. The wrapper that is used to launch the steam client adds all paths from the "old" runtime to LD_LIBRARY_PATH, which makes library lookup work there.

I did some more testing by replacing proton with a wrapper that adds the runtime directories to LD_LIBRARY_PATH, which gets me a lot further, until:

No cached sticky mapping in ActivateActionSet.
pressure-vessel-adverb[636]: Failed to open file '/proc/636/task/636/children': No such file or directory

which I assume is caused in some way by the double-bwrap setup.

I did also figure out why we get

error while loading shared libraries: libstdc++.so.6: cannot open shared object file: No such file or directory

for all pressure-vessel binaries: LD_PRELOAD contains the steam overlay gameoverlayrenderer.so. If I clear LD_PRELOAD for each command in _v2-entry-point, that goes away.

Full logs and all modifications here: https://gist.github.com/moben/12ba6e8c4735e347f1cde92c17f5183d

Most of this would of course have to be worked around on the NixOS side. I think a workable way would be to add a tmpfs at /nix/store/glibc<...>/etc/ inside the outer bwrap and write an ld.so.conf and ld.so.cache there to include the fake FHS and all other relevant directories.

@smcv Maybe it makes sense for pressure vessel to launch proton with an LD_LIBRARY_PATH containing all the runtime libdirs, as the host ld-linux.so in general can't be expected to know the debian multiarch layout? Or is this already supposed to happen? I did dumb env from my proton wrapper at some point and at least in some invocations, those were missing from LD_LIBRARY_PATH.

[L-as]

@moben mounting a tmpfs at that path with bwrap would mean that /nix/store can't be mounted to /nix/store, and instead each path inside /nix/store has to be mounted individually, and then the glibc one has to be special cased, since it would need a bind for each file/directory inside that path, so that we can make a tmpfs at the path you propose. It is possible, but it isn't trivial, and bwrap can't even handle that many binds at the moment unfortunately.

[L-as]

Actually, nevermind, it is possible since /nix/store/...-glibc-.../etc already exists: it only has a file named rpc. You wouldn't need what I described.

[smcv]

Maybe it makes sense for pressure vessel to launch proton with an LD_LIBRARY_PATH containing all the runtime libdirs, as the host ld-linux.so in general can't be expected to know the debian multiarch layout?

The host ld-linux.so doesn't need to know the Debian multiarch layout, as long as it knows the /etc/ld.so.cache path that is hard-coded in upstream glibc, and looks there for a { SONAME => path } map. We had assumed that things that are hard-coded in upstream glibc are more or less portable between glibc systems.

Unfortunately for this approach, your glibc doesn't read /etc/ld.so.cache, because it has been patched to read /nix/<a path we can't predict>/etc/ld.so.cache instead...

In the longer term we want to become less reliant on LD_LIBRARY_PATH, and more reliant on generating our own /etc/ld.so.cache (from the list of paths that we know we want to search), because games have a habit of messing with the LD_LIBRARY_PATH in ways that happen to work on typical systems but fail horribly if the LD_LIBRARY_PATH is the only way you can find libraries. That might also be a bit faster.

I think what we would need to do for that to work on NixOS is to find your glibc in /nix, traverse upwards until we get to its ${prefix}, mount a tmpfs on ${prefix}/etc, and make ${prefix}/etc/ld.so.cache a symlink to /etc/ld.so.cache. Or, you could maybe try doing the same in your outer bwrap?

[moben]

fyi, with the changes in the linked PR I only need the --filesystem=/nix workaround.

Now /etc/ld.so.conf and /etc/ld.so.cache both exist and I use the outer bwrap to create a tmpfs in the the unpredictable NixOS specific location where glibc looks for these and symlink them to /etc. Additionally, ldconfig is wrapped to write/read the files in /etc.

Not really sure if that gets me further, as the proton script seems to not launch anymore but it greatly reduces the amount of errors: https://gist.github.com/moben/e95d828095cf7adf96811c35006995ed

bwrap: execvp /usr/lib/pressure-vessel/from-host/bin/pressure-vessel-adverb: No such file or directory

everything else seems to launch okay though, so I assume there is some issue propagating the necessary mounts to the inner bwrap, so libc ends up broken in there. Or the adverb binary actually doesn't exist.

Is it supposed to create 7(?) temporary SteamLinuxRuntime_soldier/var/tmp-* chroots during a single run?

[lhindir]

Any breakthroughs on this? Thanks!

[smcv]

Please be patient, and in the meantime, use Proton 5.0 or older on NixOS. Solving this is very likely to need changes in both the NixOS packaging and pressure-vessel.

I have a half-finished pressure-vessel branch that will probably help this (and Exherbo, and other non-FHS OS layouts), but it isn't the highest-priority topic that I have to work on.

Is it supposed to create 7(?) temporary SteamLinuxRuntime_soldier/var/tmp-* chroots during a single run?

No. If it's creating any temporary chroots in var/tmp-*, then it's got into a mode that might become the default later, but wasn't really meant to be the default yet. The idea is that instead of relying on LD_LIBRARY_PATH to force games to find the right libraries, we make a "cheap" copy of the runtime (using hardlinks or btrfs reflinks to avoid actually duplicating file content), then go through the copy and delete all the libraries that are older versions of the ones we got from the host system, so that the game has no choice but to use the ones we wanted it to use.

In that mode, there is meant to be exactly one copy of the runtime in var/ at any given time: before creating a new copy, pressure-vessel-wrap deletes old copies that are no longer in use. However, if the game is failing to start up (as it does on NixOS, because something breaks our assumptions), then it might be leaving a process running that holds a lock on the old copy, meaning that we detect the old copy as still in use, and don't delete it.

pressure-vessel was originally designed to wrap a single command, like bwrap - we had to bolt on support for using a single container for multiple commands in order to support Proton games, and the interface between Steam and pressure-vessel isn't ideal.

Each attempt at running a game is meant to create one container session, which in the case of Proton/Wine games is shared between zero or more setup commands (installing DLLs into the Wine prefix) and the actual game. I suspect that what you're seeing is:

• setup command 1 tries to start the container, fails, and leaves a pressure-vessel process running
• setup command 2 does the same
• ...
• setup command 6 does the same
• the actual game does the same

(because if a setup command fails, Steam carries on to the next one anyway - it doesn't give us a way to tell it that we failed to set up the container and it should give up now).

[smcv]

One thing that might be helpful would be to try to run something simpler in the container: instead of running a complete Proton game, try to just run an xterm. When I'm back at work next week I'll put together a suitable command to try.

[smcv]

Please could someone try this, in either the SteamLinuxRuntime or SteamLinuxRuntime_soldier directory?

LD_DEBUG=files PRESSURE_VESSEL_VERBOSE=1 ./run xterm

It will produce a lot of output, but that output should help us to see what is going wrong, and will hopefully point us towards a solution or workaround, either in pressure-vessel or in NixOS.

Or it might be easier if someone can explain step-by-step how to get from https://nixos.org/download.html#nixos-iso or https://nixos.org/download.html#nixos-virtualbox to being able to see the failure occur. I am not familiar with how NixOS works, and I can't dedicate enough time and space to this to be able to install all the operating systems people might want to use pressure-vessel on, but if I can use a virtual machine as a shortcut then that might be enough.

[bbigras]

I did with the SteamLinuxRuntime_soldier folder.

steam.log

[davidak]

/bin/bash : mauvais interpréteur: No such file or directory

Yeah, /bin/bash don't exist in NixOS. Use the portable version /usr/bin/env bash instead.

[smcv]

If you are making a FHS environment exist for Steam's benefit anyway, I would recommend making sure /bin/bash (and /bin/sh, and other commonly-hard-coded paths) exist there. Even if pressure-vessel doesn't need that path, games that run in the LD_LIBRARY_PATH runtime will very frequently hard-code it.

If you change the #!/bin/bash scripts in SteamLinuxRuntime_soldier and SteamLinuxRuntime_soldier/pressure-vessel/bin to #!/usr/bin/env bash, does that get you further?

I'm mostly interested in having @moben run the command I posted, because they seem to have a suitably large tower of workarounds to be closer to being able to find the root cause. If other people are not using the same configuration as @moben then there's a risk that we will be spending time debugging things that @moben already knows about, which is time that I cannot really afford to spend.

[kimat]

@smcv, I ran this which gave me this before replacing shebangs and this after replacing the shebangs.

@smcv, here are also some step by step instructions to setup a nixos vm with proton 5.13 : https://gist.github.com/kimat/cf0f1a36302e5d992ef681f04cf33c5e

[moben]

@davidak /bin/bash, /bin/sh and a lot of other things exist inside the FHS environment that steam runs in. The No such file errors are just what happens if ld.so.cache doesn't exist/contain the necessary libraries and you try to run the bash from the soldier runtime. See the linked draft PR.

@smcv Thanks a lot for your time and explanations. Here is a log of running LD_DEBUG=files PRESSURE_VESSEL_VERBOSE=1 ./run xterm:

https://gist.github.com/moben/54283ad25fe94fa111f535ce92b5709a

Ends again with bwrap: execvp /usr/lib/pressure-vessel/from-host/bin/pressure-vessel-adverb: No such file or directory

Makes sense as that does not exist on the FHS host, but I think it's again some dynamic loader issue. (I'm running this in an FHS environment that is identical to the one steam itself is running in).

Interestingly, I managed to get two other errors in my own testing recently: When running a game with soldier and Proton 5.13 from within steam I got the same error as above a while ago. I added bind mounts for /* to /run/host/*, as it seemed to try and use the dynamic loader from there for capture-libs? This gets me a lot further: https://gist.github.com/moben/8f54dccc7e02658b756e0fc90b0abd67

capture-libs actually runs but complains that all the absolute paths inside the fake FHS point outside /run/host: x86_64-linux-gnu-capsule-capture-libs: warning: "/nix/store/jq7ikqkazzazydfgghia6n1zww1nnbiq-libglvnd-1.3.2/lib/libEGL.so.1.1.0" is not within prefix "/run/host" (and so on)

I actually almost get to run Spelunky 2 this way but of course all the drivers are missing...

I also tried to do some testing with the debugging tools mentioned on https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/tree/master/pressure-vessel#debugging but I always run into (pressure-vessel-wrap:590): pressure-vessel-CRITICAL **: export_contents_of_run: assertion `g_file_test ("/.flatpak-info", G_FILE_TEST_EXISTS)' failed

btw, that check in https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/master/pressure-vessel/wrap.c#L312 looks backwards, or the comment above is wrong? It bails out if that does not exist but the comment says that it won't work in flatpak.

No new workarounds except mounting the FHS env also at /run/host. I think most of the current issues stem from the non-default mode that I seem to be getting into. Or is it supposed to use/expect the dynamic loader in /run/host?

I suspect pressure-vessel correctly detects that it's running i bwrap but this FHS environment isn't flatpak based and somewhere bwrap and flatpak are conflated.

I'll try investigating why the test xterm command fails while running pressure-vessel from within steam somewhat works.

[smcv]

Thanks, I'll try to have a look at the logs next week.

Because pressure-vessel-wrap is entering a container that rearranges the filesystem, the error messages can get a bit confusing: it's not always obvious whether the absolute path you see is meant to exist inside the container, outside the container or both. In your case there's an extra layer of container, adding to the confusion.

I added bind mounts for / to /run/host/, as it seemed to try and use the dynamic loader from there for capture-libs?

/run/host is not normally meant to exist in the environment from which you ran pressure-vessel-wrap. It is meant to exist in the environment created by pressure-vessel-wrap, and contains your /usr, /bin, /sbin and /lib*.

I think part of the problem might be that pressure-vessel is assuming that because it sees a library path /nix/... on the outside, it should be using /run/host/nix/... inside, which is wrong, because /run/host only contains /usr, /bin, /sbin and /lib*. In the next release this part should be fixed, because we merged https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/merge_requests/173 (for the benefit of Vulkan layers in $HOME or graphics drivers in /opt, but it should help you too).

/usr/lib/pressure-vessel/from-host/bin/pressure-vessel-adverb: No such file or directory

This is not meant to exist outside pressure-vessel's container. It is meant to exist inside. You can inspect it by considering what would happen if the temporary copy in var/tmp-*/ was the root filesystem, with your /usr and so on mounted under /run/host.

In the mode that creates a temporary copy in var/tmp-*, the pressure-vessel directory ends up hard-linked into the container's /usr/lib/pressure-vessel/from-host (it's set up like that to make it more likely that it can work when we do https://github.com/flatpak/flatpak/issues/3797); in the mode that doesn't, it ends up mounted at /run/pressure-vessel/from-host (which Flatpak probably won't let us do).

I'm still not entirely sure why that mode is active for you, but for cross-referencing with the source code and wrapper scripts, it's activated by --copy-runtime-into or environment variable PRESSURE_VESSEL_COPY_RUNTIME_INTO. It does have the benefit that you can look at the copy in var/tmp-*

I also tried to do some testing with the debugging tools mentioned on https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/tree/master/pressure-vessel#debugging

Sorry, that document might well be out of date - most of the text is from early development before the public release of the scout container runtime, which was itself quite a while before soldier.

You will probably need to run those command-lines from your FHS environment, and add at least --filesystem=/nix to all of them, for the same reason you needed it before.

Starting a container with a copy of the host system instead of a runtime is meant to still work, but Steam never does that and we don't regularly test it, so it isn't surprising if there have been regressions. The logic for whether we're in Flatpak or not does seem wrong there. Using --runtime=/path/to/SteamLinuxRuntime_soldier/soldier/files is more likely to work.

[bonkmaykrQ]

Ubuntu w/ KDE is having issues as well. Many users are having issues with 5.13, whether it be due to Pressure Vessel, Steam Runtime, etc. I would say this is more of a generic steam issue than an issue with just NixOS.

[smcv]

Many users are having issues with 5.13, whether it be due to Pressure Vessel, Steam Runtime, etc. I would say this is more of a generic steam issue than an issue with just NixOS.

No, please do not mix up non-NixOS issues with this issue report.

The switch from the scout LD_LIBRARY_PATH runtime to the soldier container for Proton 5.13 changes a lot, and there are several regressions as a result - some of them on all OSs, some of them on specific OSs, and some that depend on the precise details of the system configuration. That does not mean that all those regressions are the same bug, or that they all have the same root cause!

Mixing up different regressions just makes it harder to keep track of what is fixed and what is still pending. The harder you make it, the more time we will have to spend triaging and checking the status of various issues, which is time that we cannot spend on actually fixing things.

[cirno-999]

Things are moving in container runtime https://gitlab.steamos.cloud/steamrt/steamrt/-/wikis/Soldier-release-notes#container-runtime Preparatory work towards being runnable on NixOS

[bonkmaykrQ]

@smcv You make a good point. I don't use NixOS anyway so I'll leave it to you guys 👌

[smcv]

In the latest beta version of SteamLinuxRuntime_soldier, we automatically do the equivalent of --filesystem=/nix, so you shouldn't need to do that any more.

Starting a container with a copy of the host system instead of a runtime is meant to still work, but Steam never does that and we don't regularly test it, so it isn't surprising if there have been regressions.

This will only matter to people like @moben who are setting up unusual situations for debugging, but there was indeed a bug here, which we fixed recently. I think it's fixed in the current beta (if not, it'll be in the next).

[jbalme]

@smcv

So, I found out Nix ships it's own Scout runtime, and managed to fix a similar issue with that runtime. The runtime is used by setting the STEAM_RUNTIME environment variable.

Will there be a similar way to use a modified Soldier runtime, or will Steam only use Soldier as shipped through Steam?

[smcv]

The price NixOS pays for not being compatible with other distros is ... it's not compatible with (binaries like Steam that are compiled for) other distros. I have some sympathy for NixOS' goals and design principles, but you can't really say you didn't bring this on yourselves!

I do want to get pressure-vessel working everywhere, but I'm afraid NixOS is quite far down the priority list, because the differences between the distros that do work (Ubuntu, Fedora, Arch, etc.) are much smaller than the differences between all those and NixOS.

https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/merge_requests/182 will maybe help, but that branch doesn't work yet. After landing a fixed version of that, we would still need to know a path where we could put a ld.so.cache so that NixOS' glibc would read it. If NixOS' glibc could be made to read a particular constant, hard-coded filename - preferably /etc/ld.so.cache, so that it's identical to the Debian, Red Hat and Arch families of distros - then that would help a lot. Even if it doesn't normally exist on NixOS systems, having glibc try it first and see if it exists would be enough, I think. Or, if that's unacceptable to NixOS' glibc maintainers: if there was a parallel build of glibc that does read /etc/ld.so.cache, and the FHS container that @moben mentioned could be made to use that glibc instead of the normal one, then that could also be a route towards success.

Alternatively, when I teach pressure-vessel to be able to work from within a Flatpak environment (which is a higher priority for me, because I suspect Steam-on-Flatpak has more users than Steam-on-NixOS and it's a more "normal" FHS environment), using the unofficial Steam package from Flathub could be a route to use Steam on NixOS with fewer NixOS-specific oddities.

So, I found out Nix ships it's own Scout runtime

Sorry, Valve can't really support this. If it happens to work, good luck! but the only environment that's really feasible to support is putting Valve's official scout LD_LIBRARY_PATH runtime (the one with run.sh) on top of some sort of FHS environment (either a FHS-compatible real host OS, or a more-FHS-compatible container on NixOS like the one @moben has referred to).

Will there be a similar way to use a modified Soldier runtime, or will Steam only use Soldier as shipped through Steam?

I can't guarantee anything either way. I'm hoping there will eventually be a way to run Proton on unofficial runtimes that claim to be soldier-compatible, similar to the way it's possible to tell Steam to use unofficial builds of Proton, because that would make my life easier when testing new versions of soldier - but at the moment that isn't possible, and the only way I can test new versions is to overwrite parts of the "real" soldier runtime.

If this becomes possible, then Valve will not be able to provide any support for it, but if it works, great.

Do bear in mind, though, that the entire point of these containers is to present a predictable interface to the games, so that each game doesn't need to be able to run on all of Ubuntu, Fedora, Arch, NixOS, and so on. The more an unofficial container runtime diverges from soldier, the less likely it is that games will still work.

[Atemu]

If NixOS' glibc could be made to read a particular constant, hard-coded filename

We read /etc/ld-nix.so.preload, I'm pretty sure we could do something similar for .cache but I don't know much about the implications of that. cc @edolstra

I'm pretty sure we couldn't load the regular file because Nix can also run on all of those Distros you mentioned and must not accidentally pull in the distro's libraries.

if there was a parallel build of glibc that does read /etc/ld.so.cache, and the FHS container that @moben mentioned could be made to use that glibc instead of the normal one, then that could also be a route towards success.

That might actually be trivial to support, I'll have a go at that later.

when I teach pressure-vessel to be able to work from within a Flatpak environment (which is a higher priority for me, because I suspect Steam-on-Flatpak has more users than Steam-on-NixOS and it's a more "normal" FHS environment), using the unofficial Steam package from Flathub could be a route to use Steam on NixOS with fewer NixOS-specific oddities.

With that working, it might even fix our FHS container because it also uses bubblewrap under the hood.

[jonringer]

@smcv is there way for you all to publish what libraries and library versions you expect to be present in the bwrapped namespace? And then allow for a way to disable containerization and inherit from the environment? E.g. PRESSURE_VESSEL_SKIP_BWRAP=1

The goals of NixOS's bwrap'ing and the pressure_vessel bwrap'ing seem to align a lot on the final desired state, but differ on assumptions about the start state; and this seems to be causing the most pain.

Implementing and maintaining the steam runtime dependencies should be a lot easier for NixOS, as we are not opinionated about what should be present in the wrapped environment, and can essentially "manifest" dependencies or an environment from nothing. Current FHS logic can be found: https://github.com/NixOS/nixpkgs/blob/master/pkgs/games/steam/fhsenv.nix

[smcv]

I'm currently on vacation, taking a much-needed break from Steam. I haven't forgotten about NixOS, and there are several changes in the pipeline that could help - but resources are limited, so please be patient.

is there way for you all to publish what libraries and library versions you expect to be present in the bwrapped namespace? And then allow for a way to disable containerization and inherit from the environment?

Not in a way that's supportable in the long term. A large part of the purpose of this work is to make the runtime environment for each game be something that is under the control of Valve and the game vendor, and can be changed in a game update without having to go round all the major and minor distributions doing coordination (that's not something that can scale, because there are so many games and so many distributions). We need to get away from the idea that there is a single environment that will be used by both the Steam client and every game, because if we do that, it will never be safe to update it to something newer with ABI or behaviour changes, and we'll be stuck in 2012 forever.

Right now, the only non-experimental container runtime environment is soldier, which is basically Debian 10, but has some libraries backported from newer versions, specially patched to work better in a container environment, or both. However, we want newer native Linux games or newer Proton versions to be able to run on newer runtimes, without coordination with distributions - hence pressure-vessel, which decouples the runtime versions from the host OS versions. If we can make soldier work on NixOS, then future runtimes should also work on NixOS, without additional work from either us or NixOS. There's already a CI job building a sniper runtime internally, provisionally based on Debian 11 (which hasn't even been released yet!), just to make sure that we still can - I can confidently predict that we will eventually want some native Linux games to depend on sniper, or on something else newer than soldier.

It looks as though NixOS is replicating the scout LD_LIBRARY_PATH Steam Runtime from first principles, similar to Arch Linux's "native Steam Runtime" (which is an ongoing support burden for both Valve and Arch Linux, and Valve specifically does not support it or recommend its use). Yes, in principle NixOS could repeat this process for soldier, and then again for sniper, and then again for medic or pyro or whatever we release after that, but please don't: this is absolutely not something we can provide support for, and it is not something that can scale. We need to be able to make changes to the runtime environment - some of which will be non-ABI-compatible! - without having every distribution's reinvention of the Steam Runtime regress and need distribution-specific fixes.

[jbalme]

I'm currently on vacation, taking a much-needed break from Steam. I haven't forgotten about NixOS, and there are several changes in the pipeline that could help - but resources are limited, so please be patient.

Happy holidays then!

is there way for you all to publish what libraries and library versions you expect to be present in the bwrapped namespace? And then allow for a way to disable containerization and inherit from the environment?

Not in a way that's supportable in the long term.

Understandable.

It looks as though NixOS is replicating the scout LD_LIBRARY_PATH Steam Runtime from first principles, similar to Arch Linux's "native Steam Runtime"

We do provide an option for that, but it is not the default, and I don't believe many people use it that way.

In fact, what we do by default is very similar to how pressure vessel works, we create a bwrap environment and unpack the upstream Valve Scout runtime into it, with a few updated packages because many games depend on libraries newer than what Scout provides (and a couple other hacks.)

Nix also provides a mechanism for using this bwrap environment to run non-Steam apps and games that expect a more traditional environment, but which we are unable to support because they're niche and unredistributable.

I wonder if it might be worth it in the long run to drop our bwrap environment entirely and ship a (possibly modified but hopefully upstreamable) pressure-vessel itself instead, then you'd only need to support pressure-vessel-in-pressure-vessel. That way we become a downstream consumer for pressure-vessel in the same way that we consume Scout now.

What would be the easiest way to start hacking on pressure vessel?

[jonringer]

This is all under the assumption that runtimes need to be patched for nix, which they might if they are only used during game runtime:

If all games will eventually use pressure vessel, we could eventually stop doing our own bwrapped steam, and just "nixify" (e.g. patchelf) steam + runtimes to work on NixOS.

Providing a usable and consistent base for game vendors to use is difficult on Linux, and I don't there's a community more familiar with the pains of packaging software than NixOS (or other non-FHS compliant distros).

I guess my ask is, allow some way for users to prevent binaries being pulled down at runtime. Trying to do "something for the user at runtime" is often detrimental to most nix workflows. Pre-compiled binaries don't work without some bwrapped/chroot'ed environment (or using something like patchelf to edit , and script may call a command that doesn't exist. Essentially we have to add "impurities", for it work, and then pressure_vessel is trying to remove the impurities.

I would much rather package the various scout, soldier, sniper, medic, pyro, ... runtimes and tell steam where to find it:

# pressure_vessel.toml
[runtimes]
scout = /nix/store/....-steamrt-scout-0.20201203.1
soldier = /nix/store/....-steamrt-soldier-...
sniper = /nix/store/....-steamrt-sniper-...
...

[options]
fail_on_missing_runtime = true

and then in nix, we can just call steam through a wrapper:

# steam
export PRESSURE_VESSEL_CONF=${PRESSURE_VESSEL_CONF:-/nix/store/...-pressure_vessel.toml}
exec /nix/store/...-steam/bin/steam-unwrapped "$@"

[jonringer]

On a slightly different note, wouldn't you want the libraries for soldier to include RUNPATH ${ORIGIN}, so that they prefer the co-located curated libraries?

I'm almost able load libraries even from nix:

Large ldd output

``` $ echo $PWD /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu $ find . -type f -name '*.so*' -exec patchelf --set-rpath '${ORIGIN}' {} \; $ ldd libSDL2-2.0.so.0.12.0 ldd: warning: you do not have execution permission for `./libSDL2-2.0.so.0.12.0' linux-vdso.so.1 (0x00007fff86f21000) libasound.so.2 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libasound.so.2 (0x00007f414858e000) libm.so.6 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libm.so.6 (0x00007f4148406000) libdl.so.2 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libdl.so.2 (0x00007f4148400000) libpulse-simple.so.0 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libpulse-simple.so.0 (0x00007f41483f9000) libpulse.so.0 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libpulse.so.0 (0x00007f41483a5000) libX11.so.6 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libX11.so.6 (0x00007f4148259000) libXext.so.6 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libXext.so.6 (0x00007f4148045000) libXcursor.so.1 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libXcursor.so.1 (0x00007f4148037000) libXinerama.so.1 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libXinerama.so.1 (0x00007f4148031000) libXi.so.6 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libXi.so.6 (0x00007f4147e20000) libXrandr.so.2 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libXrandr.so.2 (0x00007f4147c12000) libXss.so.1 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libXss.so.1 (0x00007f4147c0c000) libXxf86vm.so.1 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libXxf86vm.so.1 (0x00007f4147a05000) libwayland-egl.so.1 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libwayland-egl.so.1 (0x00007f41479ff000) libwayland-client.so.0 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libwayland-client.so.0 (0x00007f41479ed000) libwayland-cursor.so.0 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libwayland-cursor.so.0 (0x00007f41479e0000) libxkbcommon.so.0 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libxkbcommon.so.0 (0x00007f414799c000) libpthread.so.0 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libpthread.so.0 (0x00007f4147974000) librt.so.1 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./librt.so.1 (0x00007f4147969000) libc.so.6 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libc.so.6 (0x00007f414779c000) /nix/store/9l06v7fc38c1x3r2iydl15ksgz0ysb82-glibc-2.32/lib64/ld-linux-x86-64.so.2 (0x00007f4148809000) libpulsecommon-12.2.so => not found libcap.so.2 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libcap.so.2 (0x00007f4147791000) libpulsecommon-12.2.so => not found libdbus-1.so.3 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libdbus-1.so.3 (0x00007f4147738000) libxcb.so.1 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libxcb.so.1 (0x00007f4147708000) libXrender.so.1 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libXrender.so.1 (0x00007f41474fb000) libXfixes.so.3 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libXfixes.so.3 (0x00007f41472f4000) libffi.so.6 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libffi.so.6 (0x00007f41472e9000) libsystemd.so.0 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libsystemd.so.0 (0x00007f4147242000) libXau.so.6 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libXau.so.6 (0x00007f414703b000) libXdmcp.so.6 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libXdmcp.so.6 (0x00007f4146e34000) liblzma.so.5 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./liblzma.so.5 (0x00007f4146e0a000) liblz4.so.1 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./liblz4.so.1 (0x00007f4146dea000) libgcrypt.so.20 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libgcrypt.so.20 (0x00007f4146cca000) libbsd.so.0 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libbsd.so.0 (0x00007f4146cad000) libgpg-error.so.0 => /home/jon/.local/share/Steam/steamapps/common/SteamLinuxRuntime_soldier/soldier_0.20201203.0/files/lib/x86_64-linux-gnu/./libgpg-error.so.0 (0x00007f4146c88000) ```

If I flatten the directories, I should be able to load anything.

[smcv]

Please see https://github.com/ValveSoftware/steam-runtime/tree/master/doc and in particular https://github.com/ValveSoftware/steam-runtime/blob/master/doc/goals.md before proposing new approaches to runtimes. I appreciate the effort people are putting in here, but we have already thought about this quite a lot, and time that I spend responding to "why can't you just..." is time that I am not spending on making it work better (on NixOS or elsewhere), which doesn't help anyone.

wouldn't you want the libraries for soldier to include RUNPATH ${ORIGIN}, so that they prefer the co-located curated libraries?

tl;dr: No, indiscriminately loading every library from the runtime would break graphics drivers. We tried something equivalent to that in 2013.

We have to take the graphics drivers from the host system (or maybe a recent Flatpak runtime), because your GPU might require a recent version of Mesa, and we are planning for a future in which soldier continues to be supported for many years. We can't just take the version of Mesa from soldier, because your GPU might be too new for it; we also can't just backport the latest Mesa into soldier, because eventually it will no longer be feasible to do that backport (it will need newer supporting libraries and LLVM). Or, if you're using proprietary drivers such as NVIDIA's, we have to use the version of the user-space driver that matches the kernel-side driver, because they are not designed to be cross-compatible between versions.

Then, for each shared library libfoo in the recursive dependencies of your graphics drivers (most obviously libc, libgcc and libdrm), we have to be able to choose to use the version of libfoo from the same place as the graphics driver, or from the runtime, whichever one is newer. We cannot just use the version of libfoo from the same place as the graphics driver unconditionally, because if we did, and it was older than the one in the runtime, we would be breaking the dependencies of libraries in the runtime. We also can't use the libfoo from the runtime unconditionally, because if we did, and it was older than the one next to the graphics driver, we would be breaking the dependencies of the graphics driver.

(We also don't want to be compiling the entire runtime from first principles: we need to pick up security updates for "most" runtime libraries directly from Debian, without having to curate the whole thing ourselves for however many years it's in use. I think people might be overestimating how large the Steam Runtime development team is and how much time is available to put into each runtime or each distribution.)

I would much rather package the various scout, soldier, sniper, medic, pyro, ... runtimes and tell steam where to find it

Sorry, this is not going to happen. Valve are not going to go for a setup in which deployment of a Steam Runtime update or a new branch of the Steam Runtime gets stalled by waiting for NixOS to package it (or waiting for Debian to package it, or waiting for Arch to package it, or any other distribution - NixOS is not special here). They need to be able to deploy an updated package into the SteamLinuxRuntime_soldier depot in the Steampipe CDN, and count on it actually being used to run games by the time the next Proton update goes out.

I hope you can see why Valve would not be happy about the support burden of "NixOS is using their own different builds and won't pick up updates until a NixOS maintainer notices that a change has occurred". There are too many distributions for it to be feasible to special-case them all: the point of the Steam Runtime is to reduce the differences. If we give NixOS special treatment, then every other distribution with an unusual setup will also want special treatment, and that just can't scale.

I guess my ask is, allow some way for users to prevent binaries being pulled down at runtime.

This is the opposite of how Steam works. The Steampipe CDN is all about making sure that you're running the current binaries for each game and each compat tool (historically Proton, and now the soldier runtime as well), so that nobody needs to keep trying to support older binaries for the games and the compat tools.

[nrdxp]

To work around this problem apply this diff to ~/.steam/steam/steamapps/common/SteamLinuxRuntime_soldier/_v2-entry-point:

   #!/usr/bin/env bash

  +shift 4 
  +exec "${@}"
  +
   set -eu
   me="$(readlink -f "$0")"
   here="${me%/*}"

[jonringer]

comment moved to https://github.com/NixOS/nixpkgs/issues/108598#issuecomment-755670606

[jbalme]

Relevant draft PR from last year regarding ldconf in fhsenvs: https://github.com/NixOS/nixpkgs/pull/59595

[danieldk]

Moved to #108598 .