leeroybrun / glacier-vault-remove

Remove all archives stored inside an Amazon Glacier vault, even if you have a huge number of them.
379 stars 50 forks source link

A) Move to Boto3(decrease memory usage), B) Allow Role Authentication #25

Closed raphael-riel closed 7 years ago

raphael-riel commented 7 years ago

Move to Boto3(decrease memory usage)

With my large vaults (500k+ Archives), I was unable to delete the whole vault without running in a memory depletion. Even on a r4.2xlarge EC2 machine from AWS (61GB RAM).

With some research, it was looking like boto was leaking on each request. With the move toward boto3, the same run takes no more than ~400Mb.

~Caveat: Boto3 needed AWS's account id. It's now required in the run args.~

Allow role Authentication

I needed the script to run on an EC2 with access to Glacier via a IAM Role.

spg commented 7 years ago

LGTM

raphael-riel commented 7 years ago

I just found out I could dynamically fetch the AccountID using Boto... http://stackoverflow.com/questions/36709461/get-aws-account-id-from-boto ~I'll apply this method in a few moments.~ DONE!

leeroybrun commented 7 years ago

Looks good! Thanks a lot for your work! :)

raphael-riel commented 7 years ago

Thanks to you too for initialy sharing your work. This saved me a big headache!