Games should be launched with soft RLIMIT_NOFILE = 1024

[smcv]

Follow-up from #7956.

Your system information

• Steam client version (build number or date): beta 2021-08-04, version 1628114775
• Distribution (e.g. Ubuntu): Debian 11
• Opted into Steam client beta?: yes
• Have you checked for system updates?: yes

Please describe your issue in as much detail as possible:

Steam launches games with the RLIMIT_NOFILE resource limit's rlimit_cur (soft limit) member set to more than FD_SETSIZE (1024).

This could cause games that call select(2) to crash or hang if many file descriptors are allocated. File descriptors with numeric value FD_SETSIZE or more cannot fit in a fd_set, and attempting to add them to the mask with FD_SET will cause an out-of-bounds write.

Similarly, if a game allocates memory proportional to the number of file descriptors allowed, it could allocate a large amount of memory; or if a game loops over all file descriptors up to the soft limit, like [pressure-vessel used to do](), it will loop for longer than is necessary.

If games need more than 1024 fds, and the game developer knows that the game and the libraries it uses do not have any of these anti-patterns, then the game should raise the soft fd limit itself, like Proton's branch of Wine does, and then lower it back down to 1024 before executing subprocesses that are not under their control (e.g. after fork() but before execve()).

Steps for reproducing this issue:

1. Launch any game (I used Floating Point)
2. Inspect its rlimits with prlimit -p (I used prlimit -p $(pgrep Floating), which is valid for single-process games like this one)
3. Look at NOFILE - max number of open files
4. Expected result: SOFT column says 1024; HARD column says something OS-dependent and more than 1024, normally 4096 (legacy), 1024×1024 (Debian derivatives) or 512×1024 (non-Debian-derivatives that use systemd)
5. Actual result: NOFILE SOFT column is 2348 (HARD column is 1024×1024 as expected on my Debian system)

[smcv]

It looks as though gameoverlayrenderer.so also increases the soft RLIMIT_NOFILE by 100 per level of process hierarchy that the overlay is loaded into, which is why I got 2348 (2048 from Steam + 100 each from 3 levels of processes that LD_PRELOAD the gameoverlayrenderer.so while launching my example game).

As with games inheriting Steam's elevated RLIMIT_NOFILE, this is not necessarily a good idea.

[TTimo]

I feel like this has just about as many chances in turn at breaking games that are not prepared for a low limit of 1024 and already ride on our 2048 setting (which I'm planning to up to 8192).

If we actually find titles that crash because the fd limit is > 1024, we can add a configuration to force it down for just those.

CC @Plagman