search

denishpatel / pygmy

Pygmy: Saving AWS Bills on Standby DB Servers
5 stars 1 forks source link

Pygmy configuration struggles in the face of changing instances #50

Closed benchub closed 2 years ago

benchub commented 3 years ago

Maybe I'm doing something wrong, but from what I can tell...

  1. python manage.py refresh_all_db_instance_types will find all ec2 instances that match my filters
  2. python manage.py get_all_db_data will talk to those discovered instances and figure out information about them.

This is fine, so long as I don't remove any instances - which, sadly, we do all the time. Once an instance has been terminated in AWS, get_all_db_data will still try to talk to it. This is probably fixable by not trying to connect if the instance id is no longer present.

When looking into this, though, I also noticed that it seems re-running refresh_all_db_instance_types will try to pick up new instances, but only if the total count is different from last time. This is obviously not going to pick up situations where instances have been replaced. I would suggest either being much smarter about discovering if each instance discovered needs to be added, or just try to re-add everything, leaning on the current existence check if it doesn't.

AadiMehta commented 3 years ago

Ben, refresh_all_db_instance_types just pull all instance types of EC2 service and RDS service. Get all db data will pull actual data.

But we will check for the condition when an instance has been terminated/replaced from AWS.

denishpatel commented 3 years ago

This is resolved. @benchub can you test again?

benchub commented 2 years ago

Yeah, get_all_db_data was getting all the data, but that seems less than ideal to need to run that continually. https://github.com/benchub/pygmy/commit/010cedef4a4c97c5a892f39289220d5a2b847b1e should make that less needed anyway.