In the future, should consider not storing duplicate copies of files when the checksums are identical. Could log when files are not saved because checksums are identical to distinguish from cases when files were not saved due to a program or website issue.
jeanpaulrsoucy
commented
1 year ago
In the future, should consider not storing duplicate copies of files when the checksums are identical. Could log when files are not saved because checksums are identical to distinguish from cases when files were not saved due to a program or website issue.
Etags have issues with larger files since the Etags of multi-part uploads will not match locally calculated MD5 hashes. S3 has new options regarding calculating and requesting checksums: https://aws.amazon.com/blogs/aws/new-additional-checksum-algorithms-for-amazon-s3/
Of course, the alternative is to use the API to request the checksum of the most recent recent file, which would be simpler but less generalizable.