search

mlevit / aws-auto-cleanup

Programmatically delete AWS resources based on an allowlist and time to live (TTL) settings
MIT License
496 stars 55 forks source link

Execution log is showing resource is in whitelist, but resource is not in the temporary or permanent whitelist #105

Closed atqhg23 closed 2 years ago

atqhg23 commented 2 years ago

Describe the bug The execution log is showing that an EC2 instance is part of the whitelist (the action appears as SKIP- WHITELIST), but the instance is not part of the temporary or permanent whitelist, and is not showing up in the whitelist DynamoDB table.

To Reproduce I am not sure what caused this. This came up because a user was trying to add the instance to the whitelist from the execution log, but they were not able to since according to the exec log it was already part of the whitelist, when it was actually not.

I checked CloudTrail to see if there was any activity that updated the whitelist DynamoDB table to see if someone added the resource to the whitelist and then removed it, but no results came up. I looked for all non-read-only actions and also did another search only looking for actions specific to modifying the table items including BatchGetItem, ConditionCheckItem, GetItem, BatchWriteItem, DeleteItem, PutItem, UpdateItem.

I also checked the CloudWatch logs for the lambda function in dry-run, and the instance was not being at all in the logs, and actually the lambda function timed out as well since it went past 15 minutes, maybe this had something to do with it, but looking at the logs, the cleanup had already finished targeting instances and was now targeting security groups

It could be that this is no big deal. We have been running the cleanup for over 4 months now and this is the first time we hear about this.

Expected behavior The instance should have been marked as delete since it is older than the TTL that we set (25 days).

AWS (please complete the following information):

membra commented 2 years ago

Before @mlevit answers,

I had that same behavior when resource was part of a stack that was whitelisted. Can it be the same in your case? Is that EC2 part of a whitelisted stack by any chance?

mlevit commented 2 years ago

@membra is correct. If a CloudFormation Stack is whitelisted, all the Resources within that Stack are also whitelisted.

If you run Auto Cleanup in DEBUG mode (via serverless.yml) you should hopefully see more information regarding the EC2 instance being added to a whitelist.