bwrap requires user namespaces

[clarfonthey]

In my specific case, I use Arch Linux on a linux-hardened kernel, which disables the kernel.unprivileged_userns_clone flag. I do not think that a "enabled-by-default" is a valid reason to require this flag -- although it is enabled by default on the mainline kernel now, it still has outstanding security issues that have not been resolved, which is why I believe it should not be a hard requirement.

Copying from ValveSoftware/Proton#4289:

As mentioned in #4278, the release requires enabling the kernel.unprivileged_userns_clone kernel flag:

bwrap: No permissions to creating new namespace, likely because the kernel does not allow non-privileged user namespaces. On e.g. debian this can be enabled with 'sysctl kernel.unprivileged_userns_clone=1'.

This is not acceptable -- this flag is for a feature that has not been fully vetted for security and has been disabled in several security-conscious distros. The best info I could find is in this article: https://lwn.net/Articles/673597/

Personally, I think that requiring user namespaces gains absolutely no value for Proton or the steam runtime and should not be required.

[smcv]

You need either a kernel that will allow unprivileged users to create new user namespaces, or a setuid-root version of the bubblewrap executable (normally /usr/bin/bwrap).

With kernels where everyone can create new user namespaces (Ubuntu, Fedora, Arch Linux default kernel, and most kernel.org kernels), pressure-vessel doesn't need anything special. It can use pressure-vessel/bin/bwrap.

In distributions with Debian's kernel.unprivileged_userns_clone patch (Debian, Arch Linux linux-hardened kernel), there are two things you can do. Choose one of these:

• Set kernel.unprivileged_userns_clone = 1, which sets the kernel's behaviour to be similar to Ubuntu, Fedora and kernel.org kernels. With this set, pressure-vessel/bin/bwrap is sufficient.
• Install a setuid version of bwrap. The Debian bubblewrap package and the Arch Linux bubblewrap-suid package are suitable. This allows new user namespaces to be created in a restricted way by running /usr/bin/bwrap, which makes sure to avoid various routes through which user namespaces can be used to attack the kernel. It does not allow more general use of user namespaces. pressure-vessel will automatically use /usr/bin/bwrap if available, ignoring its own copy.

[smcv]

Personally, I think that requiring user namespaces gains absolutely no value for Proton or the steam runtime

Sorry, this is just not true; user namespaces do have value for the Steam Runtime. They are a key part of the mechanism we use to replace /usr and /lib with the Steam Runtime library stack, which gives us a route towards a more predictable and robust runtime environment than the traditional method of setting a very long LD_LIBRARY_PATH (which is easily broken by things like games that incorrectly unset the LD_LIBRARY_PATH, libraries that have an RPATH, and libraries that hard-code a path to look for plugins).

In particular, because every Linux executable hard-codes the path to ld.so, without replacing /usr and /lib like this, we cannot rely on having a newer version of glibc than the version in the oldest distribution that Steam supports. For example, Steam Runtime 2 (as used by Proton 5.13) is based on Debian 10, from 2019. We would not be able to run that on Ubuntu 18.04 (from 2018) without making use of namespaces.

The user namespace requirements for pressure-vessel are the same as for Flatpak, from which it recycles a lot of the setup code.

For more details, see the doc/ directory in this repository, or my talk on this subject at FOSDEM 2020.

[clarfonthey]

Wow, I completely misunderstood the scope of user namespaces, then. I was under the impression it was just for user permissions, not the way you're using it.

I do think that some form of setuid would be more appropriate, although isn't there a way of making all that work with chroot and bind mounts, rather than relying on bwrap? I skimmed the docs, but it's not entirely clear specifically what features are being used and what the general requirement of permissions is.

setuid bwrap seems like a bit of an excess for what's being done here, and I'm mostly wondering what the minimal set of permissions would be for something like this.

[clarfonthey]

As a side note, I should also add that one of my biggest objections was also requiring these features for Proton specifically.

I understand having to rearrange user permissions and files to make sure that native Linux executables are convinced they're running in a familiar environment. I don't think we should require extra permissions for emulating a Linux system for a tool that doesn't have to emulate a Linux system.

[smcv]

tl;dr: if we didn't use bwrap, we would have to reinvent it.

I do think that some form of setuid would be more appropriate

You're welcome to rely on a setuid system-wide bwrap (I do, on Debian), but Steam cannot install a setuid bwrap for itself, because Steam is unprivileged and runs as an ordinary user. Steam also should not install a setuid bwrap even if it could, because decisions about what you trust at an OS level should be made by your OS vendor and by you as sysadmin, not by Steam.

isn't there a way of making all that work with chroot and bind mounts, rather than relying on bwrap?

bwrap's use of user namespaces is precisely a way to get the ability to make bind-mounts while minimizing the damage it can do at a system level. It's actually using the pivot_root syscall instead of chroot, but they're conceptually similar (they both change what / means).

The ability to change the root directory is a dangerous thing to have, and is (correctly!) restricted by capabilities: the chroot(2) syscall is not allowed unless you have CAP_SYS_CHROOT, and therefore the chroot(1) command-line frontend to that syscall has the same restriction. For example, if Steam could create arbitrary chroot environments (but not have full root capabilities), it would be easy to bind-mount a crafted version of /etc/sudoers (that allows everything) over the real /etc/sudoers, and then we could just ask sudo to give us real root. We don't want that to be possible!

The way to let unprivileged processes enter chroot-like environments, without giving away full capabilities, is to have a setuid program that we can ask to do the setup for us, which is responsible for only allowing safe things (like bwrap or Debian's schroot), or to do IPC to a privileged daemon, which is equally responsible for only allowing safe things (like Canonical's snapd). This requires having a trusted, fully-privileged component at the OS level.

bwrap uses the PR_SET_NO_NEW_PRIVS prctl to prevent its child processes from getting new privileges from running a setuid program, which closes off attacks like the one described above. Creating user namespaces as an unprivileged user also closes off this attack, because the only user ID the kernel will let us map into the new user namespace is our own uid, so we cannot escalate privileges that way.

I'm mostly wondering what the minimal set of permissions would be for something like this

Bind-mounts require having CAP_SYS_ADMIN; other things that bwrap does require CAP_SYS_CHROOT, CAP_NET_ADMIN, CAP_SETUID, CAP_SETGID and CAP_SYS_PTRACE. There's no practical difference between having that set of capabilities in the initial user namespace and full root, because CAP_SYS_ADMIN and CAP_SYS_CHROOT are enough to carry out the attack described above and get full root; so to be safe to install, bwrap needs to prevent that attack somehow. On Fedora, Ubuntu, etc., a non-setuid bwrap can create a new user namespace and have those capabilities in the new user namespace without getting any capabilities in the "init namespace". Or, a setuid bwrap can drop privileges and use PR_SET_NO_NEW_PRIVS to prevent the attack.

I should also add that one of my biggest objections was also requiring these features for Proton specifically

You're correct to say that Proton is running Windows games that don't care what is in /usr. However, Proton itself consists of native Linux executables and libraries, and relies on a lot of lower-level libraries that come from the Steam Runtime - so it has many of the same issues as a native Linux game when it comes to depending on lower-level libraries.

Continuing to build Proton in the Steam Runtime 1 environment (which resembles a version of Ubuntu from 2012) is not sustainable: sooner or later, one of the components that go into Proton is going to require a library newer than that.

Building a complete, newer library stack on top of the ancient glibc version that is the newest that we can require, and patching it to look for plugins etc. in $STEAM_RUNTIME, is not really a practical or sustainable thing to do. I did look into doing that a while ago, but it would require redoing at least 5 years of distribution development, and would have a significant ongoing maintenance cost (more than a namespace-based approach). This is all open source, so everything is theoretically possible - but spending more time and money to get a less reliable result and take longer to achieve it is not exactly an attractive option, especially if we expect to need the namespaced-based approach sooner or later for native Linux games anyway.

[stsp]

user namespaces do have value for the Steam Runtime. They are a key part of the mechanism we use to replace /usr and /lib with the Steam Runtime library stack, which gives us a route towards a more predictable and robust runtime environment

Hi. AFAIK wine-6 will have all the 32bit library dependencies compiled as PE, and as the result, will not require the host multilib, and overall will not require too many libs from host. Will this help proton to stop providing its own library stack and drop the container tricks?

[clarfonthey]

Okay, I'm convinced on bwrap for games. Honestly, I think that the extremely knowledgeable and detailed explanations here are awesome, and it would be nice if there were more documentation put up to help convince people like myself, who may have already skimmed the docs and come to similar conclusions without knowing the full picture. I also should apologise for not fully reading into this before providing my arguments, but I will add that better documentation would have helped.

You're correct to say that Proton is running Windows games that don't care what is in /usr. However, Proton itself consists of native Linux executables and libraries, and relies on a lot of lower-level libraries that come from the Steam Runtime - so it has many of the same issues as a native Linux game when it comes to depending on lower-level libraries.

IMHO, the sole reason for the Steam runtime is to bypass the fact that a majority of games, despite being once-built, self-contained packages, are dynamically linked to libraries on the host system. Proton, being built and distributed by Valve as part of Steam, does not have to do this. A number of tools are already fully statically linked on Linux already because languages like go and Rust do this by default -- why can't Proton also be built statically as well? And furthermore, why does it explicitly rely on the Steam Runtime even when it's disabled?

[smcv]

I'm sorry, I can't provide equally detailed and knowledgeable explanations every time someone asks "why can't you just...", because the more time I spend doing that, the less time I can spend actually fixing things!

However, please let me assure you that we have thought about this, and we are reasonably sure that what we're doing is the best route available (or in some cases the least-bad route, where there is no good answer). Cross-distribution compatibility is not elegant, it's a maze of messy compromises, and this is the price we have to pay for Steam and Steam games working somewhat reliably, even on distributions like yours that are not specifically supported.

You don't necessarily have to take my word for it: Proton, pressure-vessel and bubblewrap are all open-source, and so is the Steam Runtime (apart from a few NVIDIA-proprietary libraries, for which I don't have access to any more source code than you do). The Steam client is not open-source, but its mechanism for adding "compatibility tools" is user-extensible - so if you think you have a better answer, it's entirely possible to fork the relevant projects, try it, and make your own Proton-like compatibility tool. People have done this, and sometimes it even works. However, the Proton team cannot provide you with any support for that, and neither can I.

once-built, self-contained packages

They are not self-contained, and they can't be self-contained. Your GPU needs user-space graphics drivers, which means that, at a minimum, even if absolutely everything else is statically-linked, games and Wine are going to need to be able to load those graphics drivers and their dependencies (which includes glibc!) dynamically.

If you want to be able to disable, bypass, modify or otherwise alter the Steam Runtime (even in unsupported ways), it also seems inconsistent to be advocating static linking, which would result in being unable to change anything in a game's dependencies, other than by the game's vendor publishing new binaries.

There are also legal reasons why game publishers prefer dynamic linking: if a game is linked to a LGPL-licensed module, most obviously glibc but also things like GTK, then the license requires it to be possible to substitute a modified version of that module. Shared linking (LGPL 2.1 §6.b, LGPL 3.0 §4.d.1) is well-understood, and pressure-vessel doesn't prevent replacing the runtime with your own modified version (we don't recommend this, and it is certainly not something we can support, but it's something that is possible to do). Providing object code so that you can re-link a statically linked game (LGPL 2.1 §6.a, LGPL 3.0 §4.d.0) is, realistically, not going to happen.

Finally, if you try it, you'll also find that building a large enough fraction of an operating system to be able to build Proton (and still being able to build that stack tomorrow, and still being able to build it next year) is a significant piece of work in its own right. We're "standing on the shoulders of giants" here by reusing Ubuntu packages for Steam Runtime v1, and Debian packages for Steam Runtime v2. Yes, we could rebuild everything as static libraries ourselves (and track security updates!), but the time we would spend doing that is time we could be spending on doing new things instead.

And furthermore, why does it explicitly rely on the Steam Runtime even when it's disabled?

I already know you're not going to like this answer, but: disabling or modifying the (traditional, LD_LIBRARY_PATH) Steam Runtime is not a supported thing to do. Yes, it is technically possible (even fairly straightforward) to do it; but the result is not expected to work reliably (or at all), and Valve will not provide you with technical support for the result.

Similarly, the obvious answer to why Proton relies on the Steam Runtime is "because otherwise, it doesn't work". The Proton 5.13 binaries need to find Steam Runtime v2 (heavily based on Debian 10) in the normal library search path. Your operating system is not Steam Runtime v2, and is not fully ABI-compatible with Steam Runtime v2; each distribution needs to make some decisions about how libraries and packages are set up, and the decisions made by Debian 10 are not the same as the decisions made by Arch Linux, which are not the same as the decisions made by Fedora, and so on.

There are ways to run Proton 5.13 without pressure-vessel, or with a different runtime environment in the container, and if you figure out how, you can try doing that. However, nobody is responsible for making that work for you. The Proton team have decided to release Proton 5.13 in a particular form, with particular requirements; if you don't like those requirements, nobody is compelling you to use Proton.

[sipaktli]

Personally, I think that requiring user namespaces gains absolutely no value for Proton or the steam runtime

Sorry, this is just not true; user namespaces do have value for the Steam Runtime. They are a key part of the mechanism we use to replace /usr and /lib with the Steam Runtime library stack, which gives us a route towards a more predictable and robust runtime environment than the traditional method of setting a very long LD_LIBRARY_PATH (which is easily broken by things like games that incorrectly unset the LD_LIBRARY_PATH, libraries that have an RPATH, and libraries that hard-code a path to look for plugins).

In particular, because every Linux executable hard-codes the path to ld.so, without replacing /usr and /lib like this, we cannot rely on having a newer version of glibc than the version in the oldest distribution that Steam supports. For example, Steam Runtime 2 (as used by Proton 5.13) is based on Debian 10, from 2019. We would not be able to run that on Ubuntu 18.04 (from 2018) without making use of namespaces.

The user namespace requirements for pressure-vessel are the same as for Flatpak, from which it recycles a lot of the setup code.

For more details, see the doc/ directory in this repository, or my talk on this subject at FOSDEM 2020.

You make the argument that you need a nuclear option such as unprivileged user namespaces to do something as basic as replacing the path of /usr and /lib. You can accomplish the exact same thing with exports and ENV variables. You are using a nuke to kill an ant.

LD_LIBRARY_PATH=, PATH=,C_INCLUDE_PATH=,CPLUS_INCLUDE_PATH=,LIBRARY_PATH=,...

PATH=$(pwd)/usr/bin:$(pwd)/usr/bin:$PATH