search

leeroybrun / glacier-vault-remove

Remove all archives stored inside an Amazon Glacier vault, even if you have a huge number of them.
379 stars 50 forks source link

even less memory usage #30

Closed rienafairefr closed 4 years ago

rienafairefr commented 7 years ago

Hey guys, any interest on reducing even more the memory usage of this thing ? As of now this script receives the json string from the inventory retrieval job as a whole, loads all of it in memory. I had a glacier vault with hundreds of thousands of archives, a >1GB job output. Very impossible to do even on a machine with a lot of ram. I wrote a small change to glacier-vault-remove that uses a queue for the multiprocessing part, and a json streaming mechanisms from the job_output (using the range parameter) so that we are not reading the whole job output all at once. https://github.com/rienafairefr/glacier-vault-remove/tree/jsonstream2 Is it something that you guys are interested in pulling in here ?

jmgarnier commented 6 years ago

For those who want to try rienafairefr/glacier-vault-remove fork using the docker image, I gave it a 5 min try and failed. Whereas leeroybrun worked out of the box

leeroybrun commented 4 years ago

Closing as this is quite old. Feel free to let us know if that's still an issue.