`java.lang.OutOfMemoryError: Direct buffer memory` during uploads

[phlax]

We are seeing the above error quite frequently during uploads

fuller trace:

WARNING: Remote Cache: Error while uploading artifact with digest 'dd662f55f250fdc85f7d649f5deb2a38995973e85edc0b36a23b12bec6a0e71c/3594364'
com.google.devtools.build.lib.remote.common.BulkTransferException: Error while uploading artifact with digest 'dd662f55f250fdc85f7d649f5deb2a38995973e85edc0b36a23b12bec6a0e71c/3594364'
    at com.google.devtools.build.lib.remote.util.RxUtils$BulkTransferExceptionCollector.onResult(RxUtils.java:91)
    at io.reactivex.rxjava3.internal.operators.flowable.FlowableCollectSingle$CollectSubscriber.onNext(FlowableCollectSingle.java:94)
    at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber.innerSuccess(FlowableFlatMapSingle.java:173)
    at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber$InnerObserver.onSuccess(FlowableFlatMapSingle.java:342)
    at io.reactivex.rxjava3.internal.operators.single.SingleDoFinally$DoFinallyObserver.onSuccess(SingleDoFinally.java:73)
    at io.reactivex.rxjava3.internal.observers.ResumeSingleObserver.onSuccess(ResumeSingleObserver.java:46)
    at io.reactivex.rxjava3.internal.operators.single.SingleJust.subscribeActual(SingleJust.java:30)
    at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
    at io.reactivex.rxjava3.internal.operators.single.SingleResumeNext$ResumeMainSingleObserver.onError(SingleResumeNext.java:80)
    at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle$ToSingle.onError(CompletableToSingle.java:73)
    at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.tryOnError(CompletableCreate.java:91)
    at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.onError(CompletableCreate.java:77)
    at com.google.devtools.build.lib.remote.util.RxFutures$OnceCompletableOnSubscribe$1.onFailure(RxFutures.java:102)
    at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1124)
    at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
        at io.grpc.Status.asRuntimeException(Status.java:535)
        ... 10 more
    Caused by: java.lang.OutOfMemoryError: Direct buffer memory
        at java.base/java.nio.Bits.reserveMemory(Unknown Source)

https://dev.azure.com/cncf/envoy/_build/results?buildId=146213&view=logs&j=bbe4b42d-86e6-5e9c-8a0b-fea01d818a24&t=00ade0d6-ffae-5af0-b5e0-e33b1813c8bc&l=201

[phlax]

following guidance from the interwebs i set:

startup --host_jvm_args=-XX:MaxDirectMemorySize=512m

altho that doesnt seep to have helped (fwiw i noticed an errant " so will remove and see if that does help after all)

[phlax]

seems like this may be more of a bazel issue - i switched to using google RBE and got a similar result

[mostynb]

I think you're right- this is happening on the client/bazel side.

[phlax]

thanks - i raised it upstream so will close here