search

TraceMachina / nativelink

NativeLink is an open source high-performance build cache and remote execution server, compatible with Bazel, Buck2, Reclient, and other RBE-compatible build systems. It offers drastically faster builds, reduced test flakiness, and specialized hardware.
https://nativelink.com
Apache License 2.0
1.17k stars 109 forks source link

Add support for compressed blob uploads #260

Open chrisstaite-menlo opened 1 year ago

chrisstaite-menlo commented 1 year ago

Currently we only support compression on the transport, add support for compressed uploads:

main/uploads/94448848-5c26-4e27-9448-50f9fea1947e/compressed-blobs/zstd/33cc97ba1cbeb6c3c542d594f3d96cf0a80f562a287c50f61783763be42586bd/79
chrisstaite-menlo commented 1 year ago

First update remote_execution.proto to v2.3 in #268 Then we can add support for the different compression types

chrisstaite-menlo commented 1 year ago

281 adds support for reading the compressed blob resource names. Now we just need to handle the compressed blob.

chrisstaite-menlo commented 1 year ago

Might be nice if these were passed through GrpcStore and then if the CompressionStore were in use it could simply used the pre-compressed content?

allada commented 1 year ago

Might be nice if these were passed through GrpcStore and then if the CompressionStore were in use it could simply used the pre-compressed content?

We can read the capabilities API in the GrpcStore, and if it supports compressed uploads/downloads use it?

blizzardc0der commented 6 months ago

I'd like to work on this issue. cc: @allada