Open ryanmacdonald opened 1 month ago
Worth noting that I see a similar error when I attempt to use Bazel v7.2.1 without the ----sandbox_tmpfs_path=/tmp
flag:
ERROR: /workspace/us/cbf/user/ryanmacdonald/BUILD:9:19: Executing genrule //pkg-preprocess failed: (Exit 1): linux-sandbox failed: error executing Genrule command
(cd /usr/local/home/ryanmacdonald/.cache/bazel/_bazel_ryanmacdonald/997005025ac9f0daba86b61b2c3d2ad0/sandbox/linux-sandbox/40/execroot/_main && \
src/main/tools/linux-sandbox-pid1.cc:320: "mount(/tmp/cloud/batch/004741035, /tmp/cloud/batch/004741035, nullptr, MS_BIND | MS_REC, nullptr)": No such file or directory
Could you share the Bazel command you are running with all its flags as well as the directory in which it runs? What's in /tmp/cloud
?
Full bazel command line:
bazel build --define proj=foo --compilation_mode=opt --remote_cache=<remote cache url> --extra_toolchains=@local_jdk//:all --sandbox_tmpfs_path=/tmp //path/to/target
/tmp/cloud/
has a batch
subdirectory and then a bunch of subdirs under that with 9 digit names:
/tmp/cloud/batch> ls
001301595 003002190 004112935 005060954 005683908 013316919 014160521 014516820 014571466 014625115
001693463 003240165 004841924 005061070 007954108 013504958 014170704 014527071 014574927 014632679
001695585 003240703 004932092 005543441 008888472 013630435 014317761 014541580 014582087 014641326
002331398 003424045 004961724 005543559 009281013 014079770 014324237 014546661 014585025 014647758
002353218 003545566 005044664 005623208 012103415 014096863 014375605 014553554 014600433
003002110 003860811 005045323 005683519 012720324 014102594 014401783 014571380 014609712
All these dirs are empty
I tried adding --spawn_strategy=processwrapper-sandbox
to these options and that seems to remove the linux-sandbox-pid1.cc
error, but now the build errors out with a Java stack overflow about 15% of the time
Do you know where these /tmp/cloud
folders come from? Bazel may mount them if you were to run the build under /tmp
, but it looks like you aren't. If these directories are updated concurrently, that could explain the failure.
Ah, I forgot to include that I'm setting our --output_base=/tmp/bazel_build_<unique_id>
as well for the above errors
These /tmp/cloud
dirs are apparently created by our runner where each /tmp/cloud/batch/<#>
dir is individually given to a job such that within the job $TMPDIR
evaluates to some unique /tmp/cloud/batch/<#>
dir
Should I be doing something like --output_base="$TMPDIR"
and --sandbox_tmpfs_path="$TMPDIR"
?
Hey @fmeum, any other thoughts on this?
You may be running into https://github.com/bazelbuild/bazel/issues/23217, albeit with a different error message. Does your build succeed without TMPDIR
set?
@oquenchil Can you take a look?
Description of the bug:
I'm seeing the following SIGBUS error signature while running a large Scala build with Bazel v6.3.0:
I see in a few different previously filed GitHub issues (e.g., here and here) have resolved this error by adding the
--sandbox_tmpfs_path=/tmp
flag, however when I do this I see:The error occurs sporadically, about 30-50% of the time in trials I've done
Which category does this issue belong to?
Core
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
N/A
Which operating system are you running Bazel on?
Red Hat EL 8.10
What is the output of
bazel info release
?release 6.3.0
If
bazel info release
returnsdevelopment version
or(@non-git)
, tell us how you built Bazel.No response
What's the output of
git remote get-url origin; git rev-parse HEAD
?If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response