search

uskudnik / amazon-glacier-cmd-interface

Command line interface for Amazon Glacier
MIT License
374 stars 100 forks source link

Partsize of 4096 causes RAM to grow excessively #97

Open echamberlain opened 11 years ago

echamberlain commented 11 years ago

Using a partsize of 4096 causes the python process on my machine to use all the available RAM.

Is it possible to not load the entire chunk into memory before upload?

wvmarle commented 11 years ago

Mmm... I should look into what happens if you use that size. That's a 4GB part size, the maximum Amazon allows. Only to be used for really really huge files.

It should mmap the file, so it's not read in memory. Only 1 MB at a time is read into memory to take the SHA hash of. However if you're on a 32-bit system mmap has a limit of 2GB. I've no idea what happens if you try to mmap 4GB in that situation.

I think one thing we should do is strongly enforce a 2GB part size limit.