Open luxe opened 1 year ago
It's likely from here because the operation expired from redis somehow: https://github.com/bazelbuild/bazel-buildfarm/blob/74f7799d08c0b6bd02a8730d873338f461a2c6be/src/main/java/build/buildfarm/instance/shard/ShardInstance.java#L2502-L2505
That NOT_FOUND was supposed to inspire the client to retry the execution (in the virtuous loop in RemoteSpawnStrategy, outside of executeRemotely). This inspiration has fallen flat, as bazel obviously sees this as a build-breaking error.
The relevant retrier and logic is here, and only currently says 'shouldRetry' for an all-NOT_FOUND BulkTransferException, and a failed precondition status that 'looks' like a retriable error. We can try to shim this response in as a FailedPrecondition to fake out the bazel client, or we can get clarity on the REAPI itself, which is currently mum on the possible status returns for waitExecution in the case of an unknown operation id, which will be required to get the remote code fixed in bazel.
Client sees the following stacktrace:
We should provide an explanation for the NOT_FOUND response to the client. It appears that the operation may not have been found by the watcher: https://github.com/bazelbuild/bazel-buildfarm/blob/74f7799d08c0b6bd02a8730d873338f461a2c6be/src/main/java/build/buildfarm/server/services/ExecutionService.java#L124-L126 However, I don't see that code path as feasible.