CPU hit on chunk start - Githubissues

cinely / mule-uploader

Stubborn HTML5 Amazon S3 uploader

http://mule-uploader.com/

MIT License

223 stars 43 forks source link

CPU hit on chunk start #31

Closed cmcfadden closed 10 years ago

cmcfadden commented 10 years ago

There's a pretty large CPU hit at the start of each chunk with the V4 mule. Has anyone looked into what's causing it? It's in cryptoJS, but seems unreasonably large given the smallish amount of data being hashed.

gabipurcaru commented 10 years ago

Yes, this happens because the V4 API requires that every chunk has SHA-256 hashes. The computation happens in a background worker, but it's still quite bad. Unfortunately there's nothing I can do about it.

cmcfadden commented 10 years ago

Before I go down the route, have you explored any other JS crypto engines? It looks like forge would be substantially faster and have consistent performance across variable chunk sizes. I might play with swapping that out - we're using mule from a gigabit connection, and use large chunk sizes (50-200meg) for better perf.

gabipurcaru commented 10 years ago

@cmcfadden I have tried several solutions, and this was the best performing. I don't really remember what I tried back then, but I don't think I tried forge, so you might as well try that.

gabipurcaru commented 10 years ago

@cmcfadden Awesome news! It seems I can bypass hashing altogether by using this: http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html

This means that:

there will be no more slowing caused by hashing
no more need for web workers
no more crazy id="mule" workarounds
I can look for a crypto library that is smaller in size, thus reducing mule's size
the code will be cleaner due to not using web workers
the data will not be hashed so there is a theoretical risk of being corrupted

I have a working prototype ATM, but I have to refactor / clean the code; I hope I will follow up with a patch soon.

cmcfadden commented 10 years ago

Very cool. Let me know when you want some testing and I'll beat on it a bit. I didn't realize you could use presigned requests for chunked upload.

gabipurcaru commented 10 years ago

Just pushed an update, feel free to abuse it and let me know if you find any bugs (file a new bug or reopen this one).