criemen
commented
1 month ago Reading the code a bit closer, I guess this is sort-of expected behavior: If we have an error connecting to the remote action cache, we just fall back to executing the action (in my case, locally), produce the outputs locally, and move on.
But what happens in this case is:
- Action A is found in the remote action cache, so it's not executed
- Action A is marked as completed (after all, it's cached)
- Action B depends on action A's output.
- Action B is not found in the Remote Cache (let's say, it just went away)
- Now we need to pull the output from action A from the remote cache, but it just went away
- As action A was completed long ago, we have no way of requesting to re-schedule it to produce its outputs locally
This is why there's a whole top-level loop around the entire bazel invocation using experimental_remote_cache_eviction_retries which retries on element-not-found errors. This mechanism wouldn't retry the DownloadTimeoutException I'm hitting here, as that's already taken care of by the in-downloader retry mechanism.
Assuming that the remote cache has a temporary/permanent outage, though, having a retry loop around the entire build invocation would be great, otherwise an outage of the remote cache will mean that all in-flight CI jobs that rely on the cache will fail with an error. The nature of a build cache should be imo that it can go away at any point in time without affecting builds, except that they now get slower.
coeuvre
Description of the bug:
In my understanding, remote cache download errors ought to not fail the build, but rather be logged as warnings.
Today, I hit the following exception
This indicates to me that when getting a timeout during prefetching data for workers, the build will fail hard, instead of the usual warning that's logged about the timeout, and bazel moving on with the build.
Getting those timeouts in the first place is problematic and something I need to investigate, but having them crash the build is not great.
We're using a HTTP cache (bazel-remote), bazel is configured with
(plus cache and credential helper specification, omitted here)
Which category does this issue belong to?
No response
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
No response
Which operating system are you running Bazel on?
Linux, Ubuntu 22.04
What is the output of
bazel info release?release 7.2.1
If
bazel info releasereturnsdevelopment versionor(@non-git), tell us how you built Bazel.No response
What's the output of
git remote get-url origin; git rev-parse HEAD?No response
If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response