Uploading to disk cache should succeed even when uploading to remote cache fails (or vice-versa)

[peakschris]

Description of the bug:

We have two use cases where we need to push existing local build outputs to a remote cache:

1) We have a set of caches around the globe, and need to push each nightly build from the (single) build server to all of the caches. The caches do not have their own synchronization. 2) If the cache goes down during a nightly build, we need to allow the nightly build to continue to success locally, and populate the cache once it comes up again.

I can't see any feature in bazel that would allow this. It would be massively helpful for us, and we could perhaps make a contribution if it could be aligned to bazel's architecture.

I'm imagining this extra option --remote_upload_existing_results=true

bazel build //... --remote_cache=... --remote_upload_local_results=true --remote_upload_existing_results=true

Behaviour with remote_upload_existing_results=true:

• if bazel chooses to short circuit an action due to the action result already existing on disk with correct checksum, the action and action result are still sent to the cache if missing.

This could be integrated into bazel or developed as a parallel tool.

Which category does this issue belong to?

Remote Execution

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

windows

What is the output of bazel info release?

7.3.1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[tjgq]

I believe an implementation of --remote_upload_existing_results (assuming it must upload all transitive actions and not just the top-level ones) would be at odds with the current architecture, which avoids inspecting the transitive closure of any actions whose outputs can be proven to be already up-to-date.

Would it work to run your nightly builds against a local --disk_cache, and have an external helper tool to populate the remote cache from the disk cache contents? (Note that you can simultaneously use --disk_cache and --remote_cache, so in normal operation your builds would still upload to the remote cache; the disk cache would only be needed to provide cache hits while the remote cache is down and to repopulate it again.)

[websteroy]

@tjgq When we run the build simultaneously with --disk_cache and --remote_cache and the remote cache is down, why doesn't bazel write new outputs to the disk cache? It writes to the disk cache only if the remote cache is up or if --remote_cache is not specified. Is that a defect?

[tjgq]

My bad - I was under the impression that the uploads to the disk and remote caches would proceed independently, but we actually abort on the first failure. That's definitely something we should fix.

[tjgq]

I'm going to repurpose this issue to fix uploading to the disk cache when the remote cache upload fails. It should be a simple fix.

[websteroy]

@tjgq That would be great.👍