search

gaul / s3proxy

Access other storage backends via the S3 API
Apache License 2.0
1.79k stars 232 forks source link

S3Proxy 2.3.0 release #677

Closed gaul closed 2 months ago

gaul commented 3 months ago

I would like to run a new release to get more testing coverage of the storage class override middleware and include a bunch of bug fixes. Unfortunately I cannot upgrade the jclouds dependency for this release. @michaelcourcy @timursaikaliev @twickline can you test 2.3.0-SNAPSHOT? Any blockers or easy fixes that I overlooked?

michaelcourcy commented 2 months ago

@gaul I retry the wiki with the 2.3.0. and no issue.

I also find interesting to create kopia snapshots with this configuration because kopia run a lot of s3 operation and is used by kasten 

kopia repository create s3 \
        --bucket=mcourcy-testia \
        --access-key=local-identity \
        --secret-access-key=local-credential \
        --endpoint=127.0.0.1:8080 \
        --disable-tls \
        --prefix=s3-proxy/

# Create  password in the input

kopia repository connect s3 \
        --bucket=mcourcy-testia \
        --access-key=local-identity \
        --secret-access-key=local-credential \
        --endpoint=127.0.0.1:8080 \
        --disable-tls \
        --prefix=s3-proxy/

# provide the password you created 

kopia snapshot create a-directory-with-plenty-files

Then I check that all kopia blobs under s3-proxy/ on the aws-s3 test-ia bucket were all having the standard-ia storage class and that was the case except for one file .storageconfig image

Which show that kopia can force the storage class over s3proxy, or maybe .storageconfig is created by an operation that is not overridden by the middleware. Some investigation is needed. But for my use it does not matter that much.

I changed one file in the a-directory-with-plenty-files and re-run 

kopia snapshot create a-directory-with-plenty-files

To experiment more GET operation and files comparison.

All went fine.

michaelcourcy commented 2 months ago

Combination of this middleware with other middleware may create the most unexpected issues. But I'm not sure it's relevant for the moment.

gaul commented 2 months ago

@michaelcourcy Can you open a separate issue for any unexpected interactions between middlewares? Perhaps reordering them is sufficient or maybe there needs to be some kind of pipeline like Swift uses.