pressure-vessel: Fall back to copying instead of hardlinks

[casKd-dev]

pressure-vessel creates hardlinked copies of the runtime from steamrt v2 (Soldier) onwards for temporary operations.

These copies can be quite heavy in cases where the filesystem is networked, causing useless wait times for small operations (cephfs on HDDs takes up to 40 minutes for setting up the hardlinks). Choosing to have ./var as a tmpfs fails because hardlinks aren't allowed across different filesystems.

The container runtime should have a option to fall back to normal copies or use overlayfs instead to reduce the amount of operations required for these operations.

[smcv]

pressure-vessel already does fall back to copying if hard-linking fails (although not if it succeeds but is merely slow).

We can't really win here: none of the options for what to do with a non-local filesystem are great. I'd recommend using an ordinary Unix filesystem like ext4, btrfs or zfs for the Steam library that contains SteamLinuxRuntime_* if at all possible; that doesn't necessarily have to be the same filesystem that contains your actual games, although I suspect games and Proton are also not really tested or supported on networked filesystems.

Falling back to copying should work (with a warning Unable to create hard link...), but it's really slow on some filesystems, which is why a var symlink is automatically removed if present (to avoid people unintentionally setting up a very slow situation and reporting bugs about it). A var bind-mount or setting the PRESSURE_VESSEL_VARIABLE_DIR environment variable should work via the copy fallback, with a warning, at your own risk (in particular don't use a world-writeable directory like /tmp if you want to be secure against symlink attacks by a malicious local user).

overlayfs isn't yet supported by bubblewrap and can have some very weird behaviours as a result of it combining two dissimilar filesystems into one, leading to being unable to create hard-links within what appears to be the same directory, and safety mechanisms like rm -r --one-file-system not working as expected because the files in question no longer count as being on the same filesystem.

[casKd-dev]

pressure-vessel already does fall back to copying if hard-linking fails

Oh, i've seemed to see multiple var/tmp.XXXXXX directories being created in the strace when i debugged this but it seems like my assumption is wrong.

I'd recommend using an ordinary Unix filesystem like ext4, btrfs or zfs for...

That's what i am currently doing right now to avoid this problem.

although I suspect games and Proton are also not really tested or supported on networked filesystems.

Most of them deal surprisingly well when there is posix feature compatibility but some seem to have a filesystem specific init like some source games with filesystem_*.so

Minding all this, wouldn't it be better to make APPID specific runtime directories like Proton does for wine prefixes? One could instead have the "expensive" copy operation just once and then work on top of that.

[smcv]

wouldn't it be better to make APPID specific runtime directories like Proton does for wine prefixes? One could instead have the "expensive" copy operation just once and then work on top of that

That would use a lot of disk space for one extra copy of soldier per installed game that you have ever run, instead of up to one temporary copy per game running right now; and we often wouldn't be able to optimize away the extra disk space and copying time by using hard-links or reflinks, because there's no guarantee that soldier is in the same Steam library as the game's compatdata directory.

That would also get us into dealing with cache invalidation, which is bad for reliability (it's often said to be one of the hardest things in computer science). The copy of soldier in var/tmp-* is not just a simple copy, it's a copy that has been edited to remove libraries that would cause incompatibilities with the ones we import from your host system - but the edits we need to make can change whenever you make changes to the installed libraries on the host system (most commonly during upgrades), so we have to detect that situation and invalidate the cached copy, with any failure to do that potentially breaking your games.

I've been thinking about some ways to avoid the need to actually copy anything by building up a very long bubblewrap command-line that separately bind-mounts every library except the ones we need to remove, but that's not going to be a quick change to make, and I'm not sure yet whether it's feasible (doing 1000+ separate bind-mounts could have its own performance problems).