Get in contact with Valve (Steam developers)

In a recent blog article, Pierre-Loup Griffais said:

The Linux landscape has changed dramatically since we released the initial version of Steam for Linux, and as such, we are re-thinking how we want to approach distribution support going forward. There are several distributions on the market today that offer a great gaming desktop experience such as Arch Linux, Manjaro, Pop!_OS, Fedora, and many others. We'll be working closer with many more distribution maintainers in the future. If you're working on such a distribution and don't feel your project has a direct line of contact with us, by all means, have a representative reach out directly.

Source: https://steamcommunity.com/app/221410/discussions/0/1640915206447625383/

E-Mail: pgriffais@valvesoftware.com

From: https://steamcommunity.com/app/221410/discussions/0/1696043806550421224/

We can use this offer to get in contact and tell them some pain points for us.

Like:

disappearing Downloads / breaking URLs
...

What else are problems we have with Steam that upstream has to fix?

Who wants to be a contact person for them?

cc @abbradar @jagajaga @tadfisher

I'm interested - at least we can discuss disappearing repository packages.

I can be a representative if everyone is okay with that.

I composed and sent an e-mail describing our general approach and asking for feedback and help regarding breaking URLs. Let's see how this goes ;)

@abbradar have you got a response? can you post the conversation here? I'm very interested!

Sadly nothing so far.

My complete e-mail, for openness sake:

Hello,

I'm one of current NixOS [1] maintainers responsible for our Steam support. We are a distribution built around Nix, a functional package manager with an unique way of doing things. Following your shout-out in [2] I wanted to reach out on behalf of other NixOS maintainers to describe how we implement Steam support and offer feedback.

In [2] you mentioned that you consider light containerization as a possible way forward for Steam - we've actually been doing this for many years already ;). Our distribution is mostly not FHS-compliant - we don't have usual /usr, /bin, /lib etc. So for running Steam we dynamically create a container which emulates an FHS distribution with packages from NixOS linked at expected places. This is completely transparent to user - they just run steam and it works. There's no measurable performance hit and practically no additional storage space is needed -- we use symlink farms for that.

We also build our own Steam Runtime with some packages replaced by our versions for better experience - for example libpulseaudio, vulkan-loader and several others. It is linked into this container and provided for Steam via STEAM_RUNTIME environment variable. Users can set package options to add more packages to the container, switch out parts of the runtime and do more customization if needed.

Finally we offer steam-run tool for our users which runs any piece of software in a container on top of Steam Runtime. This is mostly used for running games from third-party game stores.

Since our initial implementation of the container infrastructure so far Steam works great on our distribution if I can say so myself. Our main (small) pain point currently is packaging Steam Runtime. Nature of Nix requires all package sources to be completely deterministic. We have a script which generates a package using metadata from steamrt deb repository [3]. This meta-package [4] is locked onto fixed versions of all packages at a given point of time. It downloads all .deb files, unpacks and merges them. We also don't have any separation of binary packages vs their sources - all packages are provided as so-called Nix expressions which are somewhat similar to package descriptions in other distributions. Instead we use caching: the package manager hashes the full package description and checks if there is a prebuilt package with this hash on our binary cache service which continuously builds packages. Finally, we have a policy of not caching unfree software - because of license concerns and because it doesn't bring any benefits. The package manager will transparently "build" such a package for a user which for unfree software usually means it'll just download and extract several binaries from official websites.

That brings us to the pain point itself - when upstream (Valve) updates packages in steamrt repository they immediately remove older versions of these packages, which makes them inaccessible for our users trying to install Steam. Currently we solve this with mirroring, and the one who operates the mirror (me) updates it from time to time. It'd be great if old versions of packages were too available in the upstream repository - it would ease maintenance on us because anyone could then regenerate the package and push the update without worrying about a mirror. Another way would be to use a pre-built versioned Steam Runtime available somewhere, with old versions also retained. Maybe there is actually one?

That concludes the description of our current state of the art. I'm interested in a general feedback to our approach and maybe some ideas concerning old steamrt packages. Allow me also to say "thank you" to the great team who made Linux gaming such a pleasurable experience. Keep up the good work!

1: https://nixos.org/ 2: https://steamcommunity.com/app/221410/discussions/0/1640915206447625383/ 3: http://repo.steampowered.com/steamrt/ 4: https://github.com/NixOS/nixpkgs/blob/master/pkgs/games/steam/runtime-generated.nix

-- Nikolay.

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the related code and @ mention them in a comment.
Ask on the NixOS Discourse.
Ask on the #nixos channel on irc.freenode.net.

This has already happened in https://github.com/NixOS/nixpkgs/pull/114024

For reference, our contact for issues regarding pressure-vessel is @smcv who has always been very helpful and kind :)

For reference, our contact for issues regarding pressure-vessel is @smcv

Please don't rely on contacting me personally: I am not a Valve employee, I will not necessarily be maintaining pressure-vessel forever, and I am not always available. Instead, please contact https://github.com/ValveSoftware/steam-runtime/issues for the Steam Runtime (including pressure-vessel, SteamLinuxRuntime, scout, soldier and sniper), https://github.com/ValveSoftware/Proton/issues for Proton issues not related to the Steam Runtime, or https://github.com/ValveSoftware/steam-for-linux/issues/ for Steam itself.

You will probably find https://gitlab.steamos.cloud/steamrt/steam-runtime-tools/-/blob/main/docs/distro-assumptions.md useful. A lot of what's in that document was prompted by my discussions with NixOS developers.

We also build our own Steam Runtime with some packages replaced by our versions for better experience - for example libpulseaudio, vulkan-loader and several others.

Sorry, this is something that is specifically not supported. The only supported way to run Steam itself is currently to use the LD_LIBRARY_PATH Steam Runtime, and the only supported way to use the LD_LIBRARY_PATH runtime is to use the specific build of it that is bundled with Steam.

Normally, packages in Valve's Steam Runtime are transparently ignored in favour of newer or equal versions from "the OS", so if you install both 32- and 64-bit versions of Vulkan-Loader, they will be used in preference to Valve's versions, as long as they are an equal or newer version. If Valve's version is strictly newer, it is a functional requirement that we must use it, because if your older version has strictly fewer symbols than Valve's, then games that have already started to refer to those newer symbols will not start. I say "the OS" but on NixOS, what I really mean is the FHS-style bubblewrap container that you run Steam inside, which is effectively behaving like a FHS-style OS from Steam's point of view. The logic for this is in steam-runtime/setup.sh.

It's not clear to me whether this mechanism is fully working on NixOS, because the way we find the newer versions relies on ld.so.cache and ldconfig, which are used in typical distributions like Ubuntu or Fedora, but generally not used in NixOS. I gave some advice on this when we were getting the newer container runtime working: basically, if you generate a ld.so.cache in your FHS-style container, so that running /sbin/ldconfig -XNv inside the container will list all of the FHS-style container's libraries (the same way it would on Ubuntu or Fedora), then the LD_LIBRARY_PATH Steam Runtime should work as intended, and prefer your FHS-style container's libraries over its own.

Similarly, if you install both 32- and 64-bit versions of libpulse in the FHS-style container, then Valve's version of libpulse should be detected as older and ignored. The same applies to almost every library: there are only a few exceptions for which we have to "pin" Valve's version because of historical ABI issues.

We have a script which generates a package using metadata from steamrt deb repository

The apt repository intentionally only carries the packages that are in the current general-availability version and the current beta. This is how apt repositories normally work: continuing to carry old packages in an apt repository after they have been obsoleted is very unusual.

If what you want is a stable location to download old Steam Runtime binaries with fixed hash/contents, use https://repo.steampowered.com/steamrt-images-scout/snapshots/ which has a numbered subdirectory for every version we have released to the public since 2020 (we might start to clean up old versions one day, but I expect we'll probably always keep at least a year's worth of releases). The version-numbered directories correspond to ubuntu12_32/steam-runtime/version.txt in Steam, and each one contains a file named steam-runtime.tar.xz with the same content as Steam's ubuntu12_32/steam-runtime/.

steamrt-images-heavy has a similar structure, but corresponds to ubuntu12_64/steam-runtime-heavy/version.txt and ships as steam-runtime-heavy.tar.xz with the same content as Steam's ubuntu12_64/steam-runtime-heavy/.

Sorry, this is something that is specifically not supported. The only supported way to run Steam itself is currently to use the LD_LIBRARY_PATH Steam Runtime, and the only supported way to use the LD_LIBRARY_PATH runtime is to use the specific build of it that is bundled with Steam.

I just checked and we actually still do this. I thought we had removed the runtime from our FHSEnv but apparently we haven't. This is certainly something we want to move away from; especially now that Steam has pressure-vessel.

What about native games though? I've had at least one instance where the game crashed on NixOS but worked with the Steam runtime enabled (Talos principle VR).
Is there a way to make Steam run everything via its runtime by default (without modifying user state)? It's a problem already and I'm not sure that'd get any better if we used our own libs rather than using our pinned steam-runtime's.

the way we find the newer versions relies on ld.so.cache and ldconfig, which are used in typical distributions like Ubuntu or Fedora, but generally not used in NixOS. I gave some advice on this when we were getting the newer container runtime working: basically, if you generate a ld.so.cache in your FHS-style container, so that running /sbin/ldconfig -XNv inside the container will list all of the FHS-style container's libraries (the same way it would on Ubuntu or Fedora), then the LD_LIBRARY_PATH Steam Runtime should work as intended, and prefer your FHS-style container's libraries over its own.

This is great to know, thank you!

I've been seeing ldconfig errors in the Steam log and we've observed that bundled SDL2s' mechanism to switch themselves out with a newer version from the environment isn't working as expected, so I believe this is currently not working.

It's not clear to me however whether or not we want Steam to use our newer libraries over the runtime's in all cases. We move fast (a rolling channel similar to Fedora Rawhide and a stable one similar to regular Fedora releases) and wouldn't want to break peoples' gaming setups.

If what you want is a stable location to download old Steam Runtime binaries with fixed hash/contents, use https://repo.steampowered.com/steamrt-images-scout/snapshots/

Note that the original issue is from 2019. We have been using the snapshots since 2020: https://github.com/NixOS/nixpkgs/commit/593c28b88617241a5ca69986a6aa5f57581e247a

What about native games though? I've had at least one instance where the game crashed on NixOS but worked with the Steam runtime enabled (Talos principle VR).

Steam runtime, or Steam runtime? There are two ways a typical native Linux game (compiled for the scout environment) can run:

scout LD_LIBRARY_PATH runtime ($STEAM_RUNTIME)
scout-on-soldier container runtime ("Steam Linux Runtime")

We need better/clearer names for these - the naming conventions have evolved over time and don't necessarily make as much sense as they ideally would.

If you don't force the use of "Steam Linux Runtime", the majority of native Linux games run in the traditional LD_LIBRARY_PATH runtime.

If you use Properties -> Compatibility -> Force the use of -> Steam Linux Runtime, then that's a per-game opt-in to the container runtime.

Both runtimes rely on ldconfig and ld.so.cache, but especially the LD_LIBRARY_PATH one.

We are starting to see a few games that either default to the scout-on-soldier container runtime (Dota 2) or require the sniper container runtime, which is newer than scout (Retroarch and a non-default branch of Wesnoth). Older/unmaintained games will continue to be scout binaries which run in one of the two ways I linked above, so we have to at least semi-support that setup essentially forever. I hope that in future, we'll get to a situation where all new games and most actively-updated games are set up to run in a sniper or newer container (like Retroarch does), but we are not there yet.

It is possible that we might eventually switch all surviving scout binaries to run in the scout-on-soldier container runtime instead of the LD_LIBRARY_PATH runtime, but I don't think we will want to do that until/unless the wider ecosystem forces us into it - we don't want to make old games regress, so we would probably only consider that change if it looks like it will fix more games than it breaks.

Is there a way to make Steam run everything via its runtime by default (without modifying user state)?

For most native Linux games built for the scout environment, Steam defaults to the scout LD_LIBRARY_PATH runtime (as modified by $STEAM_RUNTIME if you set it, but that's unsupported), and can be switched to the scout-on-soldier container runtime via Properties -> Compatibility. I don't think there is currently a way to make it switch to preferring the scout-on-soldier container runtime for all games globally. If you think there should be, a feature request in https://github.com/ValveSoftware/steam-for-linux/issues would be the way to show Valve that distros and userswant that.

I think the Steam Deck might already make more use of the scout-on-soldier container runtime than ordinary desktop Linux does, so Steam might already have some code for this internally (but I don't have a Steam Deck or access to the internals of Steam, so I have no more insight here than you do).

For Dota 2 and maybe more games in future, the scout-on-soldier container runtime is the default (but it's possible to swap back to the scout LD_LIBRARY_PATH runtime).

For games that require sniper or another newer container runtime, like Retroarch and likely more games in future, the only option is to use that container runtime.

For Windows games running under Proton / Steam Play, the Proton version in use determines the runtime environment. Current versions of Proton (5.13 to 7.0) use the soldier container runtime, which is mechanically the same as sniper, but older. Future versions of Proton are likely to switch from soldier to sniper, to get sniper's newer libraries (I don't know a timeline for this, maybe 8.0 will make that switch, maybe not). Old versions of Proton (5.0 or older) use the scout LD_LIBRARY_PATH runtime.

It's not clear to me however whether or not we want Steam to use our newer libraries over the runtime's in all cases. We move fast (a rolling channel similar to Fedora Rawhide and a stable one similar to regular Fedora releases) and wouldn't want to break peoples' gaming setups.

For the graphics driver stack and the core libraries that they depend on (glibc, libdrm, that sort of thing), there is really no option: we have to use modern libraries, otherwise modern GPUs can't work correctly. The Steam Deck uses SteamOS 3 (derived from Arch Linux) which is a rolling release, and my personal system has Debian testing/unstable which is another rolling release, so you're not alone here!

For application-level libraries (outside the graphics driver stack) like SDL, PulseAudio and libjpeg, the handling is different depending on whether you're talking about the container runtime or the LD_LIBRARY_PATH runtime.

The container runtime avoids using libraries from the host system if it doesn't strictly need to. This is one of the reasons why the container runtime exists, in fact. If you want to minimize your dependency on host-system libraries, preferring to use the container runtime is a good way to do that.

The LD_LIBRARY_PATH runtime does not have a good way to do that: it's much simpler than the container runtime, which is why it still works in some scenarios where the container runtime doesn't, but the price it pays for that simplicity is that many of the things the container runtime does are just not feasible to implement. However, in NixOS specifically, because you're using a FHS-like bubblewrap container to run Steam in, you can have some limited control over the libraries that Steam sees by upgrading the FHS container's libraries less aggressively than the libraries used for ordinary NixOS binaries. For instance, you might choose to hold back libraries to be the same version that appears in the latest released Steam Runtime branch, or the same version that appears in the Debian release that the latest Steam Runtime branch is based on. As a hint, that's currently Steam Runtime 3 'sniper', based on Debian 11, while Steam Runtime 4 'medic' is likely to be based on Debian 12 (or maybe 13).

As a general thing, if you have a feature request or if you want to ask "can we do this? if yes, how?" or similar questions as a result of this thread, it is probably best to start an issue in one of the locations I mentioned above for each topic that can be separated from other topics.

NixOS / nixpkgs

Get in contact with Valve (Steam developers) #63837