Many download targets leads to intermittent failures due to file descriptor limits

[williewillus]

I'm poking at buck2 for a personal project. What I have right now reads a remote JSON file full of URL's and sha1 hashes to download:

https://github.com/williewillus/buck2_modding_sandbox/blob/master/minecraft_assets.bzl

If I execute the target that depends on all these downloads, I often get errors due to running into default FD limits on a typical desktop linux system (Arch)

Buck 2 should either:

1. Read and respect ulimit FD limits for download rules
2. Attempt to lift the limits somehow (might not be the most reliable if the limits have already been modified)
3. Have a different flag that controls download parallelism
4. something else?

[ndmitchell]

Do you know what the FD limits are on your system? I'm not sure how many download rules we try and execute it parallel - it might be inadvertently more than we expect because we use tokio and it might be cleverly avoiding blocking threads, and inadvertently launching too many.

I don't think lifting the externally imposed limits in Buck2 itself is a good idea. Ideally we'd never get close to the ulimit FDs.

CC @krallin who wrote other IO-limiting aspects.

[williewillus]

By default on my Arch Linux system, ulimit -n returns 1024

For completeness, here's all the limits from bash's ulimit builtin:

$ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 128059
max locked memory           (kbytes, -l) 8192
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 128059
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

[ndmitchell]

Looking at trying to fix this in #444 (needs rework, but just seeing if that concept works)

[ndmitchell]

When I ran into this, I limited the number of downloads, and the problem with file descriptors showed up in writes (I think by queueing the downloads I left more space for the materialiser to run more writes simultaneously). So I think the limiting needs to be slightly lower in the stack that I tried.

[KipHamiltons]

I am building buck2 for the first time on an M1 mac. With my (default) ulimit -n == 256, I immediately get this error:

[...] Too many open files (os error 24)

The only way I could fix this was by:

• Setting my ulimit -n absurdly high
• Restarting the buck2 daemon
• Running build with -j=1 many times, getting a bit further each time

This is a bit of a barrier for new contributors.