search

buchgr / bazel-remote

A remote cache for Bazel
https://bazel.build
Apache License 2.0
582 stars 151 forks source link

Blake3 support #710

Open AlessandroPatti opened 10 months ago

AlessandroPatti commented 10 months ago

Bazel now supports blake3 as a hashing algorithm. It'd be great if we could start using it with bazel-remote as it promises impressive perfomance boosts

mostynb commented 10 months ago

The main blocker to adding support for BLAKE3 is that we need to consider if an update to the cache directory format is required, since the current format does not explicitly say what digest function the blobs are stored in.

If it is safe to assume that we will never encounter digest collisions between SHA256 and BLAKE3, then maybe we can keep the current format unchanged. If that's the case then maybe it's OK for bazel-remote to not care what the format of on-disk blobs are.

(Aside: I think someone mentioned that BLAKE3 is only fast on amd64. Something to bear in mind.)

prestonvanloon commented 9 months ago

I think it's a safe assumption that BLAKE3 and SHA256 would not have a digest collision.

However, I think there are some assumptions that sha256 is the only hashing algorithm for bazel. Example: https://github.com/buchgr/bazel-remote/blob/0fd6a4470bdb879730a519be6fefd79193894239/server/grpc.go#L138

However, there are new fields in the protos that explain which digest function is used, so it would be trivial to add the appropriate paths for BLAKE3.

https://github.com/bazelbuild/remote-apis/blob/6c32c3b917cc5d3cfee680c03179d7552832bb3f/build/bazel/remote/execution/v2/remote_execution.proto#L1555

adincebic commented 2 months ago

I tried using blake3 with bazel-remote and it works fine when I disable remote cache compression. To be precise, i needed to set storage mode to 'uncompressed' in bazel-remote. However, if I set it to zstd than it stops working.