search

nix-community / impermanence

Modules to help you handle persistent state on systems with ephemeral root storage [maintainer=@talyz]
MIT License
1.2k stars 86 forks source link

Compiling Rust projects in a bind mount causes a "Too many open files" error #206

Open campbellcole opened 2 months ago

campbellcole commented 2 months ago

When I try to compile a reasonably large Rust project from within a bind-mounted persistence location, cargo fails with "Too many open files" so reliably that it's become nearly impossible to successfully compile any of my projects. I've found this is dramatically worse when using the mold linker, but still occurs regularly with the default linker.

In the best cases, I can compile the code by continuously restarting the build and relying on incremental compiles to slowly inch it along, and in most cases (namely tests), the build will never succeed no matter how many times I restart it.

I've tried increasing ulimit -n to a ridiculously large number (1073741816), and this still happens. From what I've read, increasing this limit to a number this large is far from a solution so I'd like to figure out the root cause of this but I'm at a loss. My guess is something cargo/rustc is doing is leaking FDs in the fuse FS.

Any advice would be greatly appreciated. I understand this may not be an issue with impermanence but I'm asking here to at least get an idea of where to look next.

wvffle commented 2 months ago

Related issue in the cargo repository with straces: https://github.com/rust-lang/cargo/issues/14437

campbellcole commented 2 months ago

I commented this in that thread, but I was able to fix this by moving all of my home folder mounts to the system config with environment.persistence."path".users.myuser. This makes impermanence use bind-mounts instead of bindfs which is also dramatically more performant.