buchgr / bazel-remote

A remote cache for Bazel
https://bazel.build
Apache License 2.0
576 stars 150 forks source link

chunk-signature in s3 proxy #725

Closed hyphennn closed 5 months ago

hyphennn commented 6 months ago

Hi, I'm trying to connect S3 to our bazel-remote service, but I find some confusing thing:

I manually delete an AC file in my local file system:

image

than i build my project again. In my undertanding, bazel-remote will query S3 to find this ac file, if found, it returns to bazel client. But it seems not: our s3 buckt has this ac file, but S3 client still re-build this action and put this ac file:

image

image

Then I download the ac file in s3 bucket. Below is the file i download from s3.

image

And below is the file in my local:

image

Compared to the file in my local, file in s3 has an unexpected 'chunk-signature'.

Then I catch the ac file that finally return to bazel client, and find it still has this 'chunk-signature'. I guess bazel client cannot recognize this, so it finally re-build this action.

But I didn't find any code to do wit this chunk-signature. Then I search history issue and find #162 mentioned this, but seems no conclusion. Maybe minio imported this?

For more information, below is the config in my test env:

image

So, my question is: is this a bug or a feature? How can i config to avoid this in order to use s3 as our fallback strategy? thx a lot.

hyphennn commented 6 months ago

Finally I find that our company's S3 may not support V4 sign well. I think this is the root cause for this issue. As I changed our initialization to v2-sign and tested it, it works well. So I submitted a PR #726 , which allows a config 's3.disable_v4_sign' to use v2 sign in s3 when use access_key as auth_method. image