search

bazelbuild / bazel-buildfarm

Bazel remote caching and execution service
https://bazel.build
Apache License 2.0
652 stars 205 forks source link

Frequent BulkTransferExceptions: computed digest does not match #594

Open thna123459 opened 3 years ago

thna123459 commented 3 years ago

We are observing frequent instances of the following BulkTransferException error in both production and in our testing environments:

ERROR: /home/xyz/benchmark/linux-java/lib_4385/BUILD.bazel:2:13: Building lib_4385/liblib_4385.jar (100 source files) failed (Exit 34): com.google.devtools.build.lib.remote.BulkTransferException
        at com.google.devtools.build.lib.remote.RemoteCache.waitForBulkTransfer(RemoteCache.java:225)
        at com.google.devtools.build.lib.remote.RemoteExecutionCache.ensureInputsPresent(RemoteExecutionCache.java:68)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$0(RemoteSpawnRunner.java:358)
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:237)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:115)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:347)
        at com.google.devtools.build.lib.exec.SpawnRunner.execAsync(SpawnRunner.java:240)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:134)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:102)
        at com.google.devtools.build.lib.actions.SpawnStrategy.beginExecution(SpawnStrategy.java:47)
        at com.google.devtools.build.lib.exec.SpawnStrategyResolver.beginExecution(SpawnStrategyResolver.java:65)
        at com.google.devtools.build.lib.rules.java.JavaCompileAction.beginExecution(JavaCompileAction.java:354)
        at com.google.devtools.build.lib.actions.Action.execute(Action.java:127)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$4.execute(SkyframeActionExecutor.java:859)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:1019)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:978)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:129)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:81)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:469)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:845)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:314)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:438)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:398)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
        Suppressed: java.io.IOException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: computed digest 0ad157ce9749757f2a25fea016a3e620ce4beef03683fb08e2ed19dbdcc31db3/262 does not match expected 063257e83e78c52b695e5a68cc10d24f8d69ac87367902f92a518449147c2e9f/131
                at com.google.devtools.build.lib.remote.ByteStreamUploader.lambda$uploadBlobAsync$1(ByteStreamUploader.java:246)
                at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:192)
                at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:179)
                at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:124)
                at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
                at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1174)

Is there a way to troubleshoot this? Is there a way to clear specific cache entries once such errors happen?

werkt commented 3 years ago

This represents a particular entry not making it into the cache, and perhaps not because of the bazel client's doing - the size (131 versus 262) is suspect as a double-entry of expected content through the digest. I'll have a look at both client and server in this case to see if there's an oversight that would double up behavior.

To verify, are you working with shard or memory instance?

thna123459 commented 3 years ago

We are working with sharded instances (sharded workers and sharded servers, with a Redis cluster of 3 primaries/3 replicas)

werkt commented 3 years ago

Some troubleshooting could be done on the client if you manage to record the grpc log for a build that exhibits this failure - that would be invaluable in assigning blame (client or server). Use --experimental_remote_grpc_log=remote.log to produce a log of all grpc requests, and that static file can be used to discern the number of requests made in the course of that upload.

werkt commented 3 years ago

You also might want to try FINER level logging (possibly only for WriteStreamObserver), assuming you can take the flood - this will record the literal writeData calls that result in handling things on the server side

werkt commented 3 years ago

608 Is a base level detection of what appears to be happening here - the error above cannot be generated meaningfully without it getting hit, and hopefully on the specific point where the violation occurs.

werkt commented 3 years ago

If nobody has seen instances of the #608 exception presented, I'm going to close this for now, expecting that a new issue will get opened with that exception presentation to try to debug this further.

werkt commented 3 years ago

662 Is the presumed fix for this. Upon significant investigation, and some great help from @thna123459 in reporting logs, I've reasoned that the problem here is multiple concurrent interactions with a single OutputStream from CASFileCache. These interactions occur because of grpc request buffering, and serializing OutputStream access within (all) Writes seems like it can be a viable solution. I have observed some interesting behaviors here regardless, and this is a large change to CASFileCache's write behavior, which is, as I've quoted before, the beating heart of all of buildfarm.

I will close this issue again when I'm satisfied that the fix works, prevents further recurrences of this, and has landed.

thna123459 commented 3 years ago

After applying the upstream fixes, we are still observing digest errors. Those should probably be reported as timeouts: remote.log

[10:41:52][Step 1/1] ERROR: /private/var/tmp/_bazel_-buildagent/79fb5de9b4a04bfbc5635a1d277a569c/external/bazel_tools/tools/jdk/BUILD:346:14: Action external/bazel_tools/tools/jdk/platformclasspath.jar failed: (Exit 34): com.google.devtools.build.lib.remote.BulkTransferException
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.remote.RemoteCache.waitForBulkTransfer(RemoteCache.java:227)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.remote.RemoteCache.download(RemoteCache.java:338)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.remote.RemoteSpawnRunner.downloadAndFinalizeSpawnResult(RemoteSpawnRunner.java:487)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:307)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.exec.SpawnRunner.execAsync(SpawnRunner.java:240)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:140)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:102)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.actions.SpawnStrategy.beginExecution(SpawnStrategy.java:47)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.exec.SpawnStrategyResolver.beginExecution(SpawnStrategyResolver.java:65)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.analysis.actions.SpawnAction.beginExecution(SpawnAction.java:331)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.actions.Action.execute(Action.java:127)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$5.execute(SkyframeActionExecutor.java:855)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:1016)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:975)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:129)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:81)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:472)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:834)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:307)
[10:41:52][Step 1/1]    at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:477)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:398)
[10:41:52][Step 1/1]    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[10:41:52][Step 1/1]    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[10:41:52][Step 1/1]    at java.base/java.lang.Thread.run(Unknown Source)
[10:41:52][Step 1/1]    Suppressed: java.io.IOException: Output download failed: Expected digest '7f6625c4a15695cbc99a54b8235a6e2ca5973222629d931a5725e45f9fdc0d22/102348070' does not match received digest '4727d17aa2ef1f83b72fc764fdf282be3cd8283ddf779d64cd9cd28f566c2212/103265574'.
[10:41:52][Step 1/1]        at com.google.devtools.build.lib.remote.util.Utils.verifyBlobContents(Utils.java:201)
[10:41:52][Step 1/1]        at com.google.devtools.build.lib.remote.GrpcCacheClient$1.onCompleted(GrpcCacheClient.java:372)
[10:41:52][Step 1/1]        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:476)
[10:41:52][Step 1/1]        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
[10:41:52][Step 1/1]        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
[10:41:52][Step 1/1]        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
[10:41:52][Step 1/1]        at com.google.devtools.build.lib.remote.util.NetworkTime$NetworkTimeCall$1.onClose(NetworkTime.java:113)
[10:41:52][Step 1/1]        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
[10:41:52][Step 1/1]        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
[10:41:52][Step 1/1]        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
[10:41:52][Step 1/1]        at com.google.devtools.build.lib.remote.logging.LoggingInterceptor$LoggingForwardingCall$1.onClose(LoggingInterceptor.java:155)
[10:41:52][Step 1/1]        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:413)
[10:41:52][Step 1/1]        at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
[10:41:52][Step 1/1]        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
[10:41:52][Step 1/1]        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:721)
[10:41:52][Step 1/1]        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[10:41:52][Step 1/1]        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[10:41:52][Step 1/1]        ... 3 more
[10:41:52][Step 1/1] . Note: Remote connection/protocol failed with: execution failed com.google.devtools.build.lib.remote.BulkTransferException
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.remote.RemoteCache.waitForBulkTransfer(RemoteCache.java:227)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.remote.RemoteCache.download(RemoteCache.java:338)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.remote.RemoteSpawnRunner.downloadAndFinalizeSpawnResult(RemoteSpawnRunner.java:487)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:307)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.exec.SpawnRunner.execAsync(SpawnRunner.java:240)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:140)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:102)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.actions.SpawnStrategy.beginExecution(SpawnStrategy.java:47)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.exec.SpawnStrategyResolver.beginExecution(SpawnStrategyResolver.java:65)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.analysis.actions.SpawnAction.beginExecution(SpawnAction.java:331)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.actions.Action.execute(Action.java:127)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$5.execute(SkyframeActionExecutor.java:855)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:1016)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:975)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:129)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:81)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:472)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:834)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:307)
[10:41:52][Step 1/1]    at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:477)
[10:41:52][Step 1/1]    at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:398)
[10:41:52][Step 1/1]    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[10:41:52][Step 1/1]    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[10:41:52][Step 1/1]    at java.base/java.lang.Thread.run(Unknown Source)
[10:41:52][Step 1/1]    Suppressed: java.io.IOException: Output download failed: Expected digest '7f6625c4a15695cbc99a54b8235a6e2ca5973222629d931a5725e45f9fdc0d22/102348070' does not match received digest '4727d17aa2ef1f83b72fc764fdf282be3cd8283ddf779d64cd9cd28f566c2212/103265574'.
[10:41:52][Step 1/1]        at com.google.devtools.build.lib.remote.util.Utils.verifyBlobContents(Utils.java:201)
[10:41:52][Step 1/1]        at com.google.devtools.build.lib.remote.GrpcCacheClient$1.onCompleted(GrpcCacheClient.java:372)
[10:41:52][Step 1/1]        at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:476)
[10:41:52][Step 1/1]        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
[10:41:52][Step 1/1]        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
[10:41:52][Step 1/1]        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
[10:41:52][Step 1/1]        at com.google.devtools.build.lib.remote.util.NetworkTime$NetworkTimeCall$1.onClose(NetworkTime.java:113)
[10:41:52][Step 1/1]        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
[10:41:52][Step 1/1]        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
[10:41:52][Step 1/1]        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
[10:41:52][Step 1/1]        at com.google.devtools.build.lib.remote.logging.LoggingInterceptor$LoggingForwardingCall$1.onClose(LoggingInterceptor.java:155)
[10:41:52][Step 1/1]        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:413)
[10:41:52][Step 1/1]        at io.grpc.internal.ClientCallImpl.access$500(ClientCallImpl.java:66)
[10:41:52][Step 1/1]        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:742)
[10:41:52][Step 1/1]        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:721)
[10:41:52][Step 1/1]        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
[10:41:52][Step 1/1]        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
[10:41:52][Step 1/1]        ... 3 more
werkt commented 3 years ago

After applying the upstream fixes, we are still observing digest errors. Those should probably be reported as timeouts:

Except they're definitely not, and the direction of these problems are actually the reverse direction: This error is occuring upon download, and there's no timeout here, you're actually receiving more content than was asked for per the expected digest. It is difficult to prove that the former error is no longer happening with this output, but I'm confident saying that this is a completely different behavior that is exhibiting this new error, specifically in the read direction, rather than the write one.

80degreeswest commented 2 years ago

I actually just noticed a similar error as well. Just for context, this happened on just a handful of tests after we turned off local fallback and forced all tests to execute remotely. So before they were just silently failing and falling back to local execution anyways. I'm guessing there is some underlying issue with the tests and not necessarily a buildfarm issue but wanted to update it here anyways since this issue was still open.

com.google.devtools.build.lib.remote.common.BulkTransferException: Error while uploading artifact with digest '6f638bfbb9c31ab2bd0ae8fafeead1c937e4044e/99376775'   at com.google.devtools.build.lib.remote.util.RxUtils$BulkTransferExceptionCollector.onResult(RxUtils.java:91)   at io.reactivex.rxjava3.internal.operators.flowable.FlowableCollectSingle$CollectSubscriber.onNext(FlowableCollectSingle.java:94)   at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber.innerSuccess(FlowableFlatMapSingle.java:173)   at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber$InnerObserver.onSuccess(FlowableFlatMapSingle.java:342)   at io.reactivex.rxjava3.internal.observers.ResumeSingleObserver.onSuccess(ResumeSingleObserver.java:46)   at io.reactivex.rxjava3.internal.operators.single.SingleJust.subscribeActual(SingleJust.java:30)   at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)   at io.reactivex.rxjava3.internal.operators.single.SingleResumeNext$ResumeMainSingleObserver.onError(SingleResumeNext.java:80)   at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle$ToSingle.onError(CompletableToSingle.java:73)   at io.reactivex.rxjava3.internal.operators.completable.CompletableFromSingle$CompletableFromSingleObserver.onError(CompletableFromSingle.java:41)   at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.tryOnError(SingleCreate.java:95)   at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onError(SingleCreate.java:81)   at com.google.devtools.build.lib.remote.util.AsyncTaskCache$1.onError(AsyncTaskCache.java:324)   at com.google.devtools.build.lib.remote.util.AsyncTaskCache$Execution.onError(AsyncTaskCache.java:198)   at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle$ToSingle.onError(CompletableToSingle.java:73)   at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapCompletable$FlatMapCompletableObserver.onError(SingleFlatMapCompletable.java:97)   at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.tryOnError(CompletableCreate.java:91)   at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.onError(CompletableCreate.java:77)   at com.google.devtools.build.lib.remote.util.RxFutures$OnceCompletableOnSubscribe$1.onFailure(RxFutures.java:102)   at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1066)   at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)   at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)   at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)   at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814)   at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203)   at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179)   at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133)   at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)   at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)   at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)   at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)   at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100)   at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)   at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)   at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)   at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)   at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100)   at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)   at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)   at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)   at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814)   at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203)   at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179)   at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133)   at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)   at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)   at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)   at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814)   at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203)   at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179)   at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133)   at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)   at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)   at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)   at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:814)   at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:203)   at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:179)   at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:133)   at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)   at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)   at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)   at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)   at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:100)   at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)   at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)   at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)   at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)   at com.google.devtools.build.lib.remote.util.RxFutures$CompletableFuture.setException(RxFutures.java:276)   at com.google.devtools.build.lib.remote.util.RxFutures$2.onError(RxFutures.java:238)   at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:117)   at io.reactivex.rxjava3.internal.operators.single.SingleUsing$UsingSingleObserver.onError(SingleUsing.java:180)   at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.tryOnError(SingleCreate.java:95)   at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onError(SingleCreate.java:81)   at com.google.devtools.build.lib.remote.util.RxFutures$OnceSingleOnSubscribe$1.onFailure(RxFutures.java:172)   at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1066)   at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)   at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1213)   at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:983)   at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:771)   at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:53)   at com.google.devtools.build.lib.remote.ByteStreamUploader$AsyncUpload$1.onClose(ByteStreamUploader.java:600)   at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:557)   at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69)   at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:738)   at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:717)   at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)   at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)   at java.base/java.lang.Thread.run(Unknown Source)   Suppressed: java.io.IOException: Error while uploading artifact with digest '6f638bfbb9c31ab2bd0ae8fafeead1c937e4044e/99376775'   at com.google.devtools.build.lib.remote.ByteStreamUploader.lambda$uploadBlobAsync$1(ByteStreamUploader.java:277)   at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:192)   at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:179)   at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:124)   ... 63 more   Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: computed digest be214f30e934e4cc942736ee2a2f77937416047b/99376775 does not match expected 6f638bfbb9c31ab2bd0ae8fafeead1c937e4044e/99376775   at io.grpc.Status.asRuntimeException(Status.java:526)   ... 10 more