search

seung-lab / cloud-files

Threaded Python and CLI client library for AWS S3, Google Cloud Storage (GCS), in-memory, and the local filesystem.
BSD 3-Clause "New" or "Revised" License
36 stars 8 forks source link

Etag validation for S3 Compatible storage / clients #90

Closed malagus closed 10 months ago

malagus commented 1 year ago

Some of S3 compatible storage / clients use different part size that are hard coded in this lib.
I my case we use Min IO and sometimes client split files to part that don't follow 1m factor. For example i get part size 16777728 and last part size is 13911034 Is there any option too disable etag validation or change logic to use head request to get first part size and use it for etag validation?

william-silversmith commented 1 year ago

As a hack, you might be able to disable etag checking by specifying a range request (e.g. start=0, end=None). The head trick is a very good tip. I may look into implementing that.