pantsbuild / pants

The Pants Build System
https://www.pantsbuild.org
Apache License 2.0
3.29k stars 631 forks source link

Backtrack for network issues as well as missing digests #16667

Open stuhood opened 2 years ago

stuhood commented 2 years ago

11331 introduced backtracking for missing digests. But with either cache_content_behavior={defer,validate}, a network issue or error can also cause the need to backtrack.

This can exhibit as either:

  1. Store accesses explicitly failing (due to unretryable errors like authentication errors, etc)
  2. Store accesses getting tarpitted in retries, and consequently slowing down local process runs (which cannot fetch their inputs)

In either {defer,validate}, we can't skip or speculate these Store accesses, because we only fall back to accessing the remote if the local store doesn't have the content, meaning that the content only exists remotely. But we should be able to backtrack to the original source of the Digest and produce it locally instead.

stuhood commented 1 year ago

We see these fairly often in pantsbuild CI, unfortunately. Will disable validate for now.