gaul / s3proxy

Access other storage backends via the S3 API
Apache License 2.0
1.72k stars 223 forks source link

S3Proxy 2.3.0 release #677

Open gaul opened 1 month ago

gaul commented 1 month ago

I would like to run a new release to get more testing coverage of the storage class override middleware and include a bunch of bug fixes. Unfortunately I cannot upgrade the jclouds dependency for this release. @michaelcourcy @timursaikaliev @twickline can you test 2.3.0-SNAPSHOT? Any blockers or easy fixes that I overlooked?

michaelcourcy commented 2 weeks ago

@gaul I retry the wiki with the 2.3.0. and no issue.

I also find interesting to create kopia snapshots with this configuration because kopia run a lot of s3 operation and is used by kasten

kopia repository create s3 \
        --bucket=mcourcy-testia \
        --access-key=local-identity \
        --secret-access-key=local-credential \
        --endpoint=127.0.0.1:8080 \
        --disable-tls \
        --prefix=s3-proxy/

# Create  password in the input

kopia repository connect s3 \
        --bucket=mcourcy-testia \
        --access-key=local-identity \
        --secret-access-key=local-credential \
        --endpoint=127.0.0.1:8080 \
        --disable-tls \
        --prefix=s3-proxy/

# provide the password you created 

kopia snapshot create a-directory-with-plenty-files

Then I check that all kopia blobs under s3-proxy/ on the aws-s3 test-ia bucket were all having the standard-ia storage class and that was the case except for one file .storageconfig image

Which show that kopia can force the storage class over s3proxy, or maybe .storageconfig is created by an operation that is not overridden by the middleware. Some investigation is needed. But for my use it does not matter that much.

I changed one file in the a-directory-with-plenty-files and re-run

kopia snapshot create a-directory-with-plenty-files

To experiment more GET operation and files comparison.

All went fine.

michaelcourcy commented 2 weeks ago

Combination of this middleware with other middleware may create the most unexpected issues. But I'm not sure it's relevant for the moment.