search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
23.2k stars 4.06k forks source link

sandboxfs for runfiles #7937

Closed Globegitter closed 1 year ago

Globegitter commented 5 years ago

Description of the problem / feature request:

This is an idea I just had today, after having spent a bit more time experimenting with sandboxfs and having experienced some "issues" with the current conecpt of the runfiles.

Would it be possible instead of having bazel create a symlink runfiles tree for a binary, as it currently does, to make use of sandboxfs to create a virtual tree which then are "real" files (i.e. not symlinks) from the binary's point of view?

To be honest I am not sure if that can even work realistically as I do not know very much on FUSE file systems, but if it does it could bring similar benefits as it is bringing when using in combination with sandboxing. If this does not make sense at all I am happy for this to get closed again.

Feature requests: what underlying problem are you trying to solve with this feature?

We have noticed some issues with the fact that the files in the runfiles tree are symlinks, that can be due to tools dereferencing symlinks and then landing outside the runfiles tree in the workspace and then leading to incorrect behaviour in certain cases. If the runfiles are not symlinks this would not happen.

Also as discussed in various places creating the runfiles tree can currently be slow when there are a lot of files to be symlinked (e.g. in the nodejs world), especially on certain platforms like macOS, so maybe sandboxfs could also improve on the performance here.

What operating system are you running Bazel on?

Elementary OS 5.0 (Ubuntu 18.04)

What's the output of bazel info release?

release 0.24.0

Have you found anything relevant by searching the web?

No.

Any other information, logs, or outputs that you want to share?

cc @jmmv

jmmv commented 5 years ago

Yes, this makes sense. I actually recently heard someone complaining that runfiles tree creation was an issue for tests because it took a really long time. sandboxfs could be applied here as well. I don't know how hard it'll be at this point though as I have never looked at this part of the codebase.

The other option is to use the runfiles library and parse the manifest instead of relying on what's on disk. (Portable tests must do this anyway for Windows.)

Globegitter commented 5 years ago

But would the sandboxfs solution then not also work for windows? I thought symlinks where the main issue here.

jmmv commented 5 years ago

It would... if sandboxfs worked for Windows :P I don't know well what the FUSE situation is there.

That said, even if sandboxfs would work for Windows, requiring its presence would be a bad idea. We shouldn't make Bazel depend on a tool that needs a kernel module / superuser privileges to be installed -- but optional support is fine.

Globegitter commented 5 years ago

Yes those are fair points. Fortunately I/we are also not concerned about windows and I am personally pretty excited about the not needing symlinks anymore and everything else that sandboxfs can offer.

tristanz commented 4 years ago

This would be a nice feature. Symlinks cause endless problems for NodeJS and Python rules. It's very hard to fix these issues because file resolution is deeply embedded in underlying libraries. rules_nodejs currently has a lot of workarounds that don't fully work:

nicolov commented 4 years ago

I just learned about a new sandbox approach based on BPF running in the Linux kernel. It's much faster than FUSE used in sandboxfs (~5% overhead rather than 80%) and still entirely managed in userspace. https://lwn.net/Articles/803890/

Globegitter commented 4 years ago

Apologies for the off-topic comment but this BPF project does sound interesting indeed and has even a very similar name to sandboxfs: https://sandfs.github.io/ :)

jmmv commented 4 years ago

The BPF alternative sounds interesting. Haven't looked at it closely and the page doesn't provide a lot of information on how this actually works / provides, so it's unclear to me if it'd be suitable for our needs. But I can believe this alternative can be much faster than FUSE. gvisor is also something worth looking into re. sandboxing. Now, the "problem" is: the 80% penalty you quote is most likely due to our code not being properly optimized yet. The FUSE overhead shouldn't be as bad as this, so I'm confident we can bring the cost down significantly---I just haven't had the time.

Separately: another approach we can pursue is to use COW files instead of symlinks to resolve the correctness (not performance) issue. This unfortunately is file system-dependent, so not sure if it's really an option on Linux with the typical file systems (it is on macOS with APFS, though).

tristanz commented 4 years ago

For me, performance is not the top issue and I'm only looking for solution that works on Linux. The critical issue is that Bazel generating symlinks breaks a large number of libraries in the Python/Node ecosystem. Right now my workaround is to to rsync -a --copy-unsafe-links the entire folder to a staging location and then run my executable from there. If anybody has a better workaround (that doesn't involve monkey patching libraries), I'd be very curious to hear it. Sandboxfs fixes things nicely during build time but the issues still crop up at runtime.

sgowroji commented 1 year ago

Hi there! We're doing a clean up of old issues and will be closing this one. Please reopen if you’d like to discuss anything further. We’ll respond as soon as we have the bandwidth/resources to do so.