twisted / txaws

Twisted-based Asynchronous Libraries for Amazon Web Services and clouds that support the AWS APIs
MIT License
32 stars 18 forks source link

Use cooperator for hashing #68

Open mithrandi opened 7 years ago

mithrandi commented 7 years ago

For medium sized objects (let's say larger than 1MB but smaller than 100MB), fitting the object data into memory all at once isn't a big deal, but blocking to calculate the hash will take long enough to significantly impact other things going on in the process, especially if multiple uploads are attempted at once. One possible easy fix for this is to feed data into the hash function in multiple chunks, with cooperator driving the process to allow other servicing other events at the same time.