Closed cmcfadden closed 10 years ago
Yes, this happens because the V4 API requires that every chunk has SHA-256 hashes. The computation happens in a background worker, but it's still quite bad. Unfortunately there's nothing I can do about it.
Before I go down the route, have you explored any other JS crypto engines? It looks like forge would be substantially faster and have consistent performance across variable chunk sizes. I might play with swapping that out - we're using mule from a gigabit connection, and use large chunk sizes (50-200meg) for better perf.
@cmcfadden I have tried several solutions, and this was the best performing. I don't really remember what I tried back then, but I don't think I tried forge, so you might as well try that.
@cmcfadden Awesome news! It seems I can bypass hashing altogether by using this: http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html
This means that:
id="mule"
workaroundsI have a working prototype ATM, but I have to refactor / clean the code; I hope I will follow up with a patch soon.
Very cool. Let me know when you want some testing and I'll beat on it a bit. I didn't realize you could use presigned requests for chunked upload.
Just pushed an update, feel free to abuse it and let me know if you find any bugs (file a new bug or reopen this one).
There's a pretty large CPU hit at the start of each chunk with the V4 mule. Has anyone looked into what's causing it? It's in cryptoJS, but seems unreasonably large given the smallish amount of data being hashed.