search

bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
22.98k stars 4.03k forks source link

hermetic linux sandbox throws error when mounting sandbox directory #14226

Open psigen opened 2 years ago

psigen commented 2 years ago

Description of the problem / feature request:

When using --experimental_use_hermetic_linux_sandbox, even a basic workspace seems to fail with the following error in almost any build action:

src/main/tools/linux-sandbox-pid1.cc:612: "mount(/home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox, /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox, nullptr, MS_BIND | MS_REC, nullptr)": No such file or directory

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I have created a minimal reproduction in the repository: https://github.com/psigen/bazel-issue-14226

Steps to reproduce:

What operating system are you running Bazel on?

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"

What's the output of bazel info release?

$ cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.6 LTS"

Have you found anything relevant by searching the web?

I have not found any relevant issues in

The closest I can find is another sandbox mount issue which seems to have a separate cause.

Any other information, logs, or outputs that you want to share?

Here is the full debug output of trying to run a build action in my test workspace:

$ bazel build //:example --sandbox_debug
INFO: Analyzed target //:example (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/pras/bazel-hermetic-sandbox/BUILD.bazel:1:11: Compiling example.cpp failed: (Exit 1): linux-sandbox failed: error executing command 
  (cd /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox && \
  exec env - \
    PATH=/home/pras/.cache/bazelisk/downloads/bazelbuild/bazel-6.0.0-pre.20211025.1-linux-x86_64/bin:/home/pras/.krew/bin:/home/pras/.nvm/versions/node/v12.18.3/bin:/home/pras/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/tmp \
  /home/pras/.cache/bazel/_bazel_pras/install/850276ae221e5ad4603a47e9b49bd43a/linux-sandbox -t 15 -w /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox -w /tmp -w /dev/shm -M / -h /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5 -D -- /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/k8-fastbuild/bin/_objs/example/example.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/_objs/example/example.pic.o' -fPIC -iquote . -iquote bazel-out/k8-fastbuild/bin -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c example.cpp -o bazel-out/k8-fastbuild/bin/_objs/example/example.pic.o)
1636050087.194696190: src/main/tools/linux-sandbox.cc:152: calling pipe(2)...
1636050087.194748260: src/main/tools/linux-sandbox.cc:171: calling clone(2)...
1636050087.195282399: src/main/tools/linux-sandbox.cc:180: linux-sandbox-pid1 has PID 13827
1636050087.195306567: src/main/tools/linux-sandbox-pid1.cc:641: Pid1Main started
1636050087.195425296: src/main/tools/linux-sandbox.cc:197: done manipulating pipes
1636050087.196284581: src/main/tools/linux-sandbox-pid1.cc:580: mount: /

1636050087.196513283: src/main/tools/linux-sandbox-pid1.cc:609: writable: /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox
src/main/tools/linux-sandbox-pid1.cc:612: "mount(/home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox, /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox, nullptr, MS_BIND | MS_REC, nullptr)": No such file or directory
1636050087.214234533: src/main/tools/linux-sandbox.cc:233: child exited normally with code 1
Target //:example failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.193s, Critical Path: 0.03s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully
ulrfa commented 2 years ago

The error is probably caused by: --sandbox_add_mount_pair "/:/"

The purpose of the hermetic sandbox is to prevent mounting too much in the sandbox.

Bazel will not consider the mount pairs as dependencies, and will not rebuild if those files are changed. So you should typically not mount files that are changing. And especially not map anything containing bazel’s output root.

Mapping the whole root would not make it more hermetic than using the normal linux-sandbox.

psigen commented 2 years ago

I used that line specifically in my example code to make the changes as minimal as possible from the non-hermetic environment.

I don't see why this should cause a failure to bind mount a completely separate folder though. The error also does not seem to go away if that line is removed.

On Fri, Nov 5, 2021, 06:37 Ulrik Falklöf @.***> wrote:

The error is probably caused by: --sandbox_add_mount_pair "/:/"

The purpose of the hermetic sandbox is to prevent mounting too much in the sandbox.

Bazel will not consider the mount pairs as dependencies, and will not rebuild if those files are changed. So you should typically not mount files that are changing. And especially not map anything containing bazel’s output root.

Mapping the whole root would not make it more hermetic than using the normal linux-sandbox.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/issues/14226#issuecomment-961787961, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVM4K7VWENJ6RGYAVZZFJ3UKOXX3ANCNFSM5HMDMHDQ .

frazze-jobb commented 2 years ago

You can't mount anything on /, since the hermetic linux sandbox creates its own root dir '/', and mounts every mount_pair inside that. You can mount --sandbox_add_mount_pair="/:oldRoot" this will mount / in /oldRoot in the hermetic linux sandbox.

psigen commented 2 years ago

Got it! If that's the case, is there a preferred strategy for loading a custom rootfs? Like a separate flag for the root mount?

(I feel like that is one of the more common use cases for the hermetic sandbox. Do I just need to mount every subdirectory?)

On Fri, Nov 5, 2021, 07:35 Fredrik Frantzen @.***> wrote:

You can't mount anything on /, since the hermetic linux sandbox creates its own root dir '/', and mounts every mount_pair inside that. You can mount --sandbox_add_mount_pair="/:oldRoot" this will mount / in /oldRoot in the hermetic linux sandbox.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/bazelbuild/bazel/issues/14226#issuecomment-961821218, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVM4K2EUUKXSSXK35T5C7LUKO6PHANCNFSM5HMDMHDQ .

frazze-jobb commented 2 years ago

We did not implement support for a custom rootfs. But I guess you could just run bazel with normal linux sandbox inside there?

(Yes, you can mount every individual directory in '/' that you need in your sandbox under / i.e. --sandbox_add_mount_pair=/usr:/usr)

psigen commented 2 years ago

Well, the normal linux sandbox does not let me use a custom rootfs as far as I know (in my actual use case, I have a rootfs exported from the docker image used in my RBE environment, and I do not want to use anything from the host). I only used my own root partition in the example. This is for developers, not CI/CD, so I do not want them to have to do extra steps to work inside the sandbox.

Maybe I can make a PR into the documentation about how mounting / will cause problems? I could see it being the first thing people try if they want to hermetically mount a rootfs generated by Yocto/Docker/etc.

Does anyone have a link to the appropriate place to update the docs for the CLI flags?

psigen commented 2 years ago

Ok, I am clearly missing something in how to use --sandbox_add_mount_pair:

I changed the entry in .bazelrc to be the following:

build --experimental_use_hermetic_linux_sandbox
build --sandbox_add_mount_pair "/usr:/usr"

And now I get the following error:

$ bazel build //:example --sandbox_debug
INFO: Analyzed target //:example (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/pras/bazel-hermetic-sandbox/BUILD.bazel:1:11: Compiling example.cpp failed: (Exit 1): linux-sandbox failed: error executing command 
  (cd /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox && \
  exec env - \
    PATH=/home/pras/.cache/bazelisk/downloads/bazelbuild/bazel-6.0.0-pre.20211025.1-linux-x86_64/bin:/home/pras/.krew/bin:/home/pras/.nvm/versions/node/v12.18.3/bin:/home/pras/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
    TMPDIR=/tmp \
  /home/pras/.cache/bazel/_bazel_pras/install/850276ae221e5ad4603a47e9b49bd43a/linux-sandbox -t 15 -w /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox -w /tmp -w /dev/shm -M /usr/bin -h /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5 -D -- /usr/bin/gcc -U_FORTIFY_SOURCE -fstack-protector -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/k8-fastbuild/bin/_objs/example/example.pic.d '-frandom-seed=bazel-out/k8-fastbuild/bin/_objs/example/example.pic.o' -fPIC -iquote . -iquote bazel-out/k8-fastbuild/bin -fno-canonical-system-headers -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c example.cpp -o bazel-out/k8-fastbuild/bin/_objs/example/example.pic.o)
1636138183.250740003: src/main/tools/linux-sandbox.cc:152: calling pipe(2)...
1636138183.250764346: src/main/tools/linux-sandbox.cc:171: calling clone(2)...
1636138183.251008707: src/main/tools/linux-sandbox.cc:180: linux-sandbox-pid1 has PID 5066
1636138183.251023670: src/main/tools/linux-sandbox-pid1.cc:641: Pid1Main started
1636138183.251090767: src/main/tools/linux-sandbox.cc:197: done manipulating pipes
1636138183.251643293: src/main/tools/linux-sandbox-pid1.cc:580: mount: /usr

1636138183.251753936: src/main/tools/linux-sandbox-pid1.cc:609: writable: /home/pras/.cache/bazel/_bazel_pras/f17e96032b46eeff7d13f6f3fa1af6a3/sandbox/linux-sandbox/5/execroot/bazel_hermetic_sandbox
1636138183.251801793: src/main/tools/linux-sandbox-pid1.cc:609: writable: /tmp
1636138183.251822521: src/main/tools/linux-sandbox-pid1.cc:609: writable: /dev/shm
1636138183.252463368: src/main/tools/linux-sandbox-pid1.cc:451: calling fork...
1636138183.252580847: src/main/tools/linux-sandbox-pid1.cc:481: child started with PID 2
src/main/tools/linux-sandbox-pid1.cc:478: "execvp(/usr/bin/gcc, 0x1883de0)": No such file or directory
1636138183.253010098: src/main/tools/linux-sandbox-pid1.cc:498: wait returned pid=2, status=0x100
1636138183.253016714: src/main/tools/linux-sandbox-pid1.cc:516: child exited normally with code 1
1636138183.292290393: src/main/tools/linux-sandbox.cc:233: child exited normally with code 1
Target //:example failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.151s, Critical Path: 0.05s
INFO: 2 processes: 2 internal.
FAILED: Build did NOT complete successfully

"execvp(/usr/bin/gcc, 0x1883de0)": No such file or directory doesn't seem to make sense. I definitely have gcc installed, and I see mount: /usr immediately above in the logs.

frazze-jobb commented 2 years ago

Yes, the message is confusing, but whats actually "No such file or directory" is the things that /usr/bin/gcc depends on. Run ldd /usr/bin/gcc and you'll see the dynamically linked things that gcc depends on.

$ ldd /usr/bin/gcc linux-vdso.so.1 => (0x00007ffe1738f000) libm.so.6 => /lib64/libm.so.6 (0x00007f8c78e01000) libc.so.6 => /lib64/libc.so.6 (0x00007f8c78a33000) /lib64/ld-linux-x86-64.so.2 (0x00007f8c79103000)

In our environment we do at the minimum this:

build --sandbox_add_mount_pair=/etc
build --sandbox_add_mount_pair=/usr
 ## symlinks in to /usr
build --sandbox_add_mount_pair=/usr/bin:/bin
build --sandbox_add_mount_pair=/usr/lib:/lib
build --sandbox_add_mount_pair=/usr/lib64:/lib64
psigen commented 2 years ago

Getting closer!

I tried this with my directories, and I ran into the following error:

build --sandbox_add_mount_pair=/usr/lib32:/lib32

Mount target '/lib32' does not exist. Bazel only supports bind mounting on top of existing files/directories. Please create an empty file or directory at the mount target path according to the type of mount source.

The exact same flag with /usr/lib64:lib64 works fine. Is there some step to specify what empty directories will be created in the sandbox to use as mount points?

frazze-jobb commented 2 years ago

Sorry for taking my time, I had not run into that issue before. But I can replicate it now.

build --sandbox_add_mount_pair=/usr/lib32:/lib32 This fails for me because /usr/lib32 does not exist build --sandbox_add_mount_pair=/usr/lib:/lib32 This fails for me because /lib32 does not exist in the original filesystem. build --sandbox_add_mount_pair=/usr/lib:/lib This works because both source and target exists in the original filesystem

I guess that in the hermetic sandbox case, we should not validate the target paths, but instead create them.

github-actions[bot] commented 1 year ago

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

psigen commented 1 year ago

So... this is definitely still a bug and we were able to replicate and had a specific path forward for a fix. Should it be marked as awaiting-bazeler?

github-actions[bot] commented 4 weeks ago

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.