Closed criemen closed 1 week ago
Reading the code a bit closer, I guess this is sort-of expected behavior: If we have an error connecting to the remote action cache, we just fall back to executing the action (in my case, locally), produce the outputs locally, and move on.
But what happens in this case is:
This is why there's a whole top-level loop around the entire bazel invocation using experimental_remote_cache_eviction_retries
which retries on element-not-found errors. This mechanism wouldn't retry the DownloadTimeoutException
I'm hitting here, as that's already taken care of by the in-downloader retry mechanism.
Assuming that the remote cache has a temporary/permanent outage, though, having a retry loop around the entire build invocation would be great, otherwise an outage of the remote cache will mean that all in-flight CI jobs that rely on the cache will fail with an error. The nature of a build cache should be imo that it can go away at any point in time without affecting builds, except that they now get slower.
Nice analysis! This is exactly what happened under the hood.
The nature of a build cache should be imo that it can go away at any point in time without affecting builds, except that they now get slower.
I agree with you. I think making --experimental_remote_cache_eviction_retries
work with other remote cache error is a valid approach.
Description of the bug:
In my understanding, remote cache download errors ought to not fail the build, but rather be logged as warnings.
Today, I hit the following exception
This indicates to me that when getting a timeout during prefetching data for workers, the build will fail hard, instead of the usual warning that's logged about the timeout, and bazel moving on with the build.
Getting those timeouts in the first place is problematic and something I need to investigate, but having them crash the build is not great.
We're using a HTTP cache (bazel-remote), bazel is configured with
(plus cache and credential helper specification, omitted here)
Which category does this issue belong to?
No response
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
No response
Which operating system are you running Bazel on?
Linux, Ubuntu 22.04
What is the output of
bazel info release
?release 7.2.1
If
bazel info release
returnsdevelopment version
or(@non-git)
, tell us how you built Bazel.No response
What's the output of
git remote get-url origin; git rev-parse HEAD
?No response
If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.
No response
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
No response