timkay / aws

Easy command line access to Amazon EC2, S3, SQS, ELB, and SDB
http://timkay.com/aws/
424 stars 123 forks source link

Checksums on put to s3 #99

Closed ericjs closed 8 years ago

ericjs commented 8 years ago

This may be a feature request if there isn't already a way to do this...but if there is, could you tell me how.

From Amazon's best practices document for S3 (https://aws.amazon.com/articles/1904) I read that data integrity can be a concern "Though unlikely for any given request, data loss or corruption in transit does occasionally occur." For this concern "Amazon S3's REST PUT operation provides the ability to specify an MD5 checksum"

"When the request arrives at S3, an MD5 checksum will be recalculated for the object data received and compared to the provided MD5 checksum. If there's a mismatch, the PUT will be failed, preventing data that was corrupted on the wire from being written into S3. At that point, you can retry the PUT."

Is there a way I can use this functionality through an aws put operation? Or am I requesting a new feature?

ericjs commented 8 years ago

Doing some searching on I site I find reference to a --md5 parameter which sounds like it does what I want, though how it works is not explicitly described (I assume it does the md5 checksum locally and passed that with the underlying PUT to cause the behavior described in the doc I quotes--as opposed to requiring the user to do the checkum and pass it). I will try this out. If this is the case, perhaps my feature request should be to add listing of this (and other) switches and arguments to the command help (i.e. the output from "aws put --help").

ericjs commented 8 years ago

I've verified that --md5 does what I want...closing this...