11331 introduced backtracking for missing digests. But with either cache_content_behavior={defer,validate}, a network issue or error can also cause the need to backtrack.
This can exhibit as either:
Store accesses explicitly failing (due to unretryable errors like authentication errors, etc)
Store accesses getting tarpitted in retries, and consequently slowing down local process runs (which cannot fetch their inputs)
In either {defer,validate}, we can't skip or speculate these Store accesses, because we only fall back to accessing the remote if the local store doesn't have the content, meaning that the content only exists remotely. But we should be able to backtrack to the original source of the Digest and produce it locally instead.
11331 introduced backtracking for missing digests. But with either
cache_content_behavior={defer,validate}
, a network issue or error can also cause the need to backtrack.This can exhibit as either:
In either
{defer,validate}
, we can't skip or speculate these Store accesses, because we only fall back to accessing the remote if the local store doesn't have the content, meaning that the content only exists remotely. But we should be able to backtrack to the original source of theDigest
and produce it locally instead.