bazelbuild / bazel

a fast, scalable, multi-language and extensible build system
https://bazel.build
Apache License 2.0
23.23k stars 4.08k forks source link

FATAL: corrupt installation: file is missing or modified. #24284

Open Hchanni opened 4 days ago

Hchanni commented 4 days ago

Description of the bug:

We're running into issues where our self hosted github runners that run Bazel are running into very odd issues of some form of :

FATAL: corrupt installation: file '/opt/runner/.cache/bazel/_bazel_runner/install/88c4bda92e58e48775c9c9582c0165d4/rules_java/AUTHORS' is missing or modified.  Please remove '/opt/runner/.cache/bazel/_bazel_runner/install/88c4bda92e58e48775c9c9582c0165d4' and try again.

I'm trying to understand what could even cause this issue, I did some looking into the stat value of the files and I could see output:

I understand that within the install base Bazel's internal directive is to change datetime to 10 years ahead to check for any tampering or modification, however I'm trying to understand is there something internally within Bazel that could potentially be touching this?

Trying to figure out exactly what could cause this, is it as simple as an access of the file or actual modification/removal only?

Access: 2024-11-12 00:21:24.354977455 +0000
Modify: 2024-11-12 00:21:24.353977479 +0000
Change: 2024-11-12 00:21:24.353977479 +0000
 Birth: 2024-11-12 00:19:43.429343078 +0000

Which category does this issue belong to?

Core

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

Sequoia Mac 15.1

What is the output of bazel info release?

No response

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

release 7.0.1

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

tjgq commented 3 days ago

The implementation of IsUntampered does this:

bool IsUntampered(const Path &path) {
  struct stat buf;
  if (stat(path.AsNativePath().c_str(), &buf)) {
    return false;
  }

  // Compare with kNearFuture, not kDistantFuture.
  // This way we don't need to worry about a potentially unreliable equality
  // check if precision isn't preserved.
  return S_ISDIR(buf.st_mode) || (buf.st_mtime > kNearFuture);
}

So there's a few different things that could trigger the check:

Access to the file alone only affects st_atime, not the st_mtime. To affect the st_mtime, something needs to either modify the file contents or manually change the timestamp. If Bazel itself was doing this, it would be considered a bug.