aws-samples / amazon-s3-checksum-verification

MIT No Attribution
10 stars 5 forks source link

only small files get checksum #1

Open stas00 opened 1 year ago

stas00 commented 1 year ago

Thank you for providing a template to get s3 to generate sha256 checksums.

I have followed your instructions at https://aws.amazon.com/blogs/storage/enabling-and-validating-additional-checksums-on-existing-objects-in-amazon-s3/ and was able to get the sha256 checksums generated on upload.

However it appears that it only works for small files. e.g. here are Properties for 2 files in the same sub-folder - one large - about 400MB and another small - a few MBs:

snapshot_30 snapshot_29

I didn't set any filters in the event creation, and I can't see that it says anywhere that this feature only works up to a certain file size.

Any ideas what needs to be done to fix this?

fordarian commented 11 months ago

Thank you for providing a template to get s3 to generate sha256 checksums.

I have followed your instructions at https://aws.amazon.com/blogs/storage/enabling-and-validating-additional-checksums-on-existing-objects-in-amazon-s3/ and was able to get the sha256 checksums generated on upload.

However it appears that it only works for small files. e.g. here are Properties for 2 files in the same sub-folder - one large - about 400MB and another small - a few MBs:

snapshot_30 snapshot_29

I didn't set any filters in the event creation, and I can't see that it says anywhere that this feature only works up to a certain file size.

Any ideas what needs to be done to fix this?

The big file was probably multipart uploaded, but in the original article, they propose to check only put and post checkboxes in the s3 event settings. Just check the "Multipart upload finished" option as well.