tjgq
commented
4 days ago Some ideas to gather more information:
- Can you reproduce this with
--spawn_strategy=standalone(i.e., does it also happen with sandboxing disabled?) - Can you inspect the contents of the sandbox left over by
--sandbox_debugand verify that the executable is present at the expected location? In particular, if it is a symlink, does the symlink dangle, or does it point to the expected file? - Since this seems to occur for at least two different rules, can you reduce it further? Say, does a minimal
genrulelike the one below also reproduce the issue? - Can you provide the full list of flags you're using, including the ones set in blazercs?
genrule(
name = "gen",
outs = ["out.txt"],
cmd = "touch $@",
)
swarren12
fmeum
joeleba
Description of the bug:
I'm trying to update a fairly complicated Bazel project from Bazel v6.5.0 to v7.x.x, but encountering strange issues. Unfortunately, I can't pinpoint exactly where the issue lies, but I believe it is in Bazel itself, rather than any of the rules being imported.
Expected behaviour: upgrading from v6.5.0 to v7.x.x "just works" Actual behaviour: the build fails due to files inside the
linux-sandboxnot being foundMore details Currently, on Bazel v6.5.0, the build reliably passes both on local development workstations and in the CI environment. Upgrading to v7.x.x causes the build to occasionally fail on development machines and much more consistently fail in CI. Unfortunately, I've been unable to reproduce in an isolated example project, and I'm not sure exactly how to go about collecting more information on the problem.
I've tried upgrading to v7.0.0, v7.1.2, v7.2.1 and v7.3.1 but they all behave the same way.
It's not always the same target that fails, but it's always roughly for the same reason, which is that a file is not found within the sandbox.
One example of this is shown below. A
bazel clean --expungewas run first, and then (an equivalent of)bazel test //... --test_tag_filters=smoke, which first failed when trying to create anijarfor ajava_importfor a file checked into version control:Running the same
bazel testcommand a second time also resulted in a failure, this time failing to runjava:A third run of
bazel testcompleted successfully.A separate example, taken from the CI exhibits a similar mode of failure; this time during running of some tests:
At first glance, this looked to be a different type of failure; however,
cating thetest.logshows:Some observations:
node) also fail.--incompatible_sandbox_hermetic_tmpin these builds (so I believe it is set to the default value oftrue?); however, from what I can tell, addingcommon --noincompatible_sandbox_hermetic_tmpto.bazelrcdoes not help (I've tried explicitly setting both values, and the problem occurs with both)--sandbox_debuggives a lot more output, but I couldn't see anything useful in thereCurrently, I'm leaning towards this being a problem with multiple processes trying to interact with the sandbox at the same time; this would explain why I'm unable to reproduce it on a small project and why it fails more consistently in CI (bigger box with more cores to run tasks in parallel). However if I add
--jobs=1the problem persists, which suggests this hypothesis is wrong.Any suggestions on what could be tried in order to further triage or resolve this issue would be appreciated.
Which category does this issue belong to?
Core
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
As mentioned, I can reproduce it fairly reliably on a large project; unfortunately, I'm yet to find a way of reproducing it on a example project. I'll keep trying though!
Which operating system are you running Bazel on?
Linux (Fedora & CentOS)
What is the output of
bazel info release?7.3.1
If
bazel info releasereturnsdevelopment versionor(@non-git), tell us how you built Bazel.n/a
What's the output of
git remote get-url origin; git rev-parse HEAD?If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
I'm unable to run
bazelisk --bisect=6.5.0..7.0.0because it attempts to revert back to v6.0.0, which is incompatible with most of the rules inMODULE.bazel:(I'm trying to work out if I can use some of the 7.0.0 pre-release candidates to narrow down when it started failing, but so far no luck.
Have you found anything relevant by searching the web?
I found a similar sounding issue here: https://github.com/bazelbuild/bazel/issues/22151; however, this affects v6.5.0 and that is the version that is working for me.
Similarly, some comments in https://github.com/bazelbuild/bazel/pull/19943 seemed relevant, but I couldn't really turn them into useful avenues of investigation.
Any other information, logs, or outputs that you want to share?
Output of `bazel mod graph`
``` (monorepo@_)
├───aspect_bazel_lib@1.42.1
│ ├───bazel_skylib@1.6.1 (*)
│ ├───platforms@0.0.10 (*)
│ └───stardoc@0.5.4
│ ├───bazel_skylib@1.6.1 (*)
│ ├───rules_java@7.3.1 (*)
│ └───rules_license@1.0.0 (*)
├───bazel_skylib@1.6.1
│ └───platforms@0.0.10 (*)
├───platforms@0.0.10
│ └───rules_license@1.0.0 (*)
├───rules_cc@0.0.9
│ └───platforms@0.0.10 (*)
├───rules_java@7.3.1
│ ├───bazel_skylib@1.6.1 (*)
│ ├───platforms@0.0.10 (*)
│ ├───rules_cc@0.0.9 (*)
│ ├───rules_license@1.0.0 (*)
│ └───rules_proto@6.0.0
│ ├───bazel_features@1.11.0 (*)
│ ├───bazel_skylib@1.6.1 (*)
│ └───rules_license@1.0.0 (*)
├───rules_jvm_external@5.3
│ ├───bazel_skylib@1.6.1 (*)
│ └───stardoc@0.5.4 (*)
├───rules_license@1.0.0
├───rules_oci@1.7.6
│ ├───aspect_bazel_lib@1.42.1 (*)
│ ├───bazel_skylib@1.6.1 (*)
│ ├───platforms@0.0.10 (*)
│ └───container_structure_test@1.16.0
│ ├───aspect_bazel_lib@1.42.1 (*)
│ ├───bazel_skylib@1.6.1 (*)
│ └───platforms@0.0.10 (*)
├───rules_pkg@0.9.1
│ ├───bazel_skylib@1.6.1 (*)
│ ├───rules_license@1.0.0 (*)
│ └───rules_python@0.34.0 (*)
└───rules_python@0.34.0
├───bazel_skylib@1.6.1 (*)
├───platforms@0.0.10 (*)
├───rules_cc@0.0.9 (*)
├───rules_proto@6.0.0 (*)
├───bazel_features@1.11.0
│ └───bazel_skylib@1.6.1 (*)
└───protobuf@24.4
├───bazel_skylib@1.6.1 (*)
├───platforms@0.0.10 (*)
├───rules_cc@0.0.9 (*)
├───rules_java@7.3.1 (*)
├───rules_jvm_external@5.3 (*)
├───rules_pkg@0.9.1 (*)
├───rules_proto@6.0.0 (*)
├───abseil-cpp@20230802.0.bcr.1
│ ├───bazel_skylib@1.6.1 (*)
│ ├───googletest@1.14.0 (*)
│ ├───platforms@0.0.10 (*)
│ └───rules_cc@0.0.9 (*)
├───googletest@1.14.0
│ ├───abseil-cpp@20230802.0.bcr.1 (*)
│ ├───platforms@0.0.10 (*)
│ └───rules_cc@0.0.9 (*)
├───upb@0.0.0-20230516-61a97ef
│ ├───abseil-cpp@20230802.0.bcr.1 (*)
│ ├───bazel_skylib@1.6.1 (*)
│ ├───platforms@0.0.10 (*)
│ ├───rules_pkg@0.9.1 (*)
│ └───rules_proto@6.0.0 (*)
└───zlib@1.3.1.bcr.3
├───platforms@0.0.10 (*)
└───rules_cc@0.0.9 (*)
```
I've tried upgrading various rules, but with no luck (and generally bringing in other difficulties!).