search

TTLabs / EvaporateJS

Javascript library for browser to S3 multipart resumable uploads
1.82k stars 207 forks source link

Why are MD5 calculations required when using v4 signature? #347

Closed mziwisky closed 7 years ago

mziwisky commented 7 years ago

If you set awsSignatureVersion to '4', and computeContentMd5 to false, you get an error Option awsSignatureVersion is 4 but computeContentMd5 is not enabled.

Why are these things coupled? I can't find s3 documentation saying that checksums are required if you use v4 signatures. I'd like to use v4 sigs with the option to forego MD5 calculation, as the latter is computationally expensive and causes my UI to lag.

mziwisky commented 7 years ago

Er, nevermind, i found the bit in the docs saying this is required. From http://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-header-based-auth.html --

Note The x-amz-content-sha256 header is required for all AWS Signature Version 4 requests. It provides a hash of the request payload. If there is no payload, you must provide the hash of an empty string.

I'll keep an eye on the other issues here discussing how to calculate the checksum w/o tanking the UI.

bikeath1337 commented 7 years ago

@mziwisky There is one known impact to performance when using checksums: The larger the part size, the longer it takes to compute the checksums. The only reported issue that blocks the UI is when developers were trying to use EvaporateJS in a web worker. So far, the issue there points to differences in how web workers work. For all cases, there is a cost of "priming the pump" for each part on startup, as well.

My recommendation would be to keep your part size between 6 and 12 MB as a balance between recoverability and calculation performance.

I have branch in progress to see if we can reduce the cost of priming the pump, fwiw.