Configurable sandbox root

[mboes]

Description of the feature request:

Add a new command-line flag making it possible to choose an alternate root for the sandbox, rather than reusing / as the root and simply remounting most of its content read-only.

Feature requests: what underlying problem are you trying to solve with this feature?

Bazel is currently not hermetic. This is because:

• rules (including built-in ones) assume the existence of utilities in /usr/bin;
• the compiler toolchains (C++, Java, Python etc) are all assumed to come from somewhere in the PATH.

This is bad for reproducibility. My build might work today, but if I upgrade my system, my build might break tomorrow or lead to different object code. Bazel can run all actions in Docker containers using a controlled base image, but that has two problems:

• massive performance overhead (0.5-0.7 seconds just to spawn a container and then delete it, per action),
• build still not hermetic: need to conservatively invalidate the entire action cache if anything changes in the Docker image.

To avoid the overhead of Docker, a solution is to use a sandbox as Bazel already does, but with an alternate root that exposes exactly what the user wants to expose to build actions. We could have a --sandbox_root=foo flag, where @foo would be the name of some external repository (e.g. a http_archive or a custom repository rule).

To avoid invalidating the entire action cache just because something in the environment changed, a solution is to use something like rules_nixpkgs to declared fine grained dependencies between parts of the environment (e.g. packages) and build targets. Nixpkgs uses a single global store for all packages, which is always assumed to be /nix/store. To avoid having to install Nixpkgs globally, we could copy the store to some/dir/nix/store as part of a repository rule, where some/dir is a local directory chosen by Bazel, then pass that folder --sandbox_root, making the Nixpkgs store available at /nix/store inside the sandbox.

Currently, it is not possible to use Nixpkgs together with Bazel without manually installing Nix globally first, before calling any bazel build command (see https://github.com/tweag/rules_nixpkgs/issues/54). Custom sandbox roots would fix this.

Implementation plan

The changes required to the linux-sandbox utility would be small: just add an extra flag to set the sandbox root, and change the logic so that if such a flag is specified, then don't blindly remount everything in the mount space like we do currently and only bind mount what was provided with --sandbox_add_mount_pair.

Where it gets trickier is enabling the user to specify an external repository name in --sandbox_root, instead of just a local path. We have to force the evaluation of the corresponding repository rule somehow, before entering the sandbox.

What operating system are you running Bazel on?

NixOS (Linux).

What's the output of bazel info release?

v0.20.0

Have you found anything relevant by searching the web?

In #1797, developers ponder using a base image providing BusyBox/ToyBox to cover all basic tools that Bazel rules tend to assume, and which currently are picked from whatever happens to be in /usr/bin. That issue also mentions https://bazel.build/designs/2016/06/02/sandboxing.html, which is the design document underpinning the current sandboxing implementation and in which @philwo says there are plans to make the sandbox root (or "base image") configurable.

cc @regnat.

[jmmv]

I think this would be very interesting to do, but how do you switch the subprocesses to run under the new "root"? And is that solution portable? chroot is restricted to root.

[philwo]

@jmmv chroot and pivot_root would work, thanks to user namespaces (we actually had that in the code once, but I removed it, when it wasn't necessary anymore: https://source.bazel.build/bazel/+/4366b4886ee908164b2b063e144f0d5edebda14f:src/main/tools/linux-sandbox-pid1.cc;drc=e219a24a30960ab5c686dd09197b5e22626e1fe0;bpv=;bpt=0;l=364)

[jmmv]

Oh that's interesting. But obviously only applicable to Linux.

[benjaminp]

We actually have a hacky, mostly undocumented implementation of feature that we use in production: https://github.com/benjaminp/bazel/commit/c5bca812736eff98fad6cffc4c0670c91d62812d See the help for --sandbox_rootfs for a idea of how to use it. We use docker export to make the rootfs.

[glukasiknuro]

@benjaminp do you have plans to merge --sandbox_rootfs change to the bazel repo?

[benjaminp]

That patch still works very well for us, but unfortunately we don't have the resources to make a formal proposal (and remove the hacks).

[psigen]

Is this issue addressed at all by this recent PR? https://github.com/bazelbuild/bazel/pull/13279

[leoluk]

Is this issue addressed at all by this recent PR? #13279

Seems like it! One can now start with an empty sandbox and use --sandbox_add_mount_pair to populate it, implementing this part of the feature request:

The changes required to the linux-sandbox utility would be small: just add an extra flag to set the sandbox root, and change the logic so that if such a flag is specified, then don't blindly remount everything in the mount space like we do currently and only bind mount what was provided with --sandbox_add_mount_pair.

There's no way to use an external repository as the root, but is that even possible?

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions[bot]]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!