seung-lab / cloud-files

Threaded Python and CLI client library for AWS S3, Google Cloud Storage (GCS), in-memory, and the local filesystem.
BSD 3-Clause "New" or "Revised" License
38 stars 8 forks source link

feat: enable md5 validation for multipart s3 downloads #50

Closed william-silversmith closed 3 years ago

william-silversmith commented 3 years ago

We used the following documents as a guide:

Gracious thank you to the authors of those articles for the code and advice.

The PR here is a bit different. As the S3 logic is particular to Amazon, it was not possible to do this in the common logic. This means the multipart code will retry while other md5 failures will not. However, should they all be retrying?

Resolves #49