noah-paige
commented
2 months ago Hello @shivalkarrahul thank you for filing this issue - I was able see the same behavior for KMS Keys and S3 Buckets but not for SQS Queues strangely, do you mind confirming the SQS Queue is not deleted on environment stack deletes?
For KMS Keys and S3 Buckets - the removal policy strategy in Cloudformation are defaulted to RETAIN for these resources and thus the delete skips are seen such as in the above scenario. After thinking a bit through the current clean up behavior I do agree that we should update data.all to default to destroying these resources (S3 Bucket + KMS Key) if the data.all user enables the switch "Delete associated AWS CloudFormation stack" from the UI
Thank you for the suggestion - I will add this issue to the backlog and align with the team on prioritization of this issue / communicate any next steps here!
Below I discuss a bit more on implementation and how to go about clean up currently:
For implementation
S3 Bucket CDK Construct now provides an option auto_delete_objects that can be specified to ensure objects are deleted from a bucket before trying to delete the bucket itself. Also, there is the option to retain specific resources when calling a cfn_client.delete_stack(...) operation so we can still provide a user a retain option if needed to avoid risk of accidental data loss/deletion
Is this similar to what you had in mind? Curious if there is a different process for which you would want data.all to handle deletions?
For manual clean up now
Cleaning up S3 and SQS can be done via console manually as long as user has correct permissions yes. For KMS Keys - it is a bit more complicated since we also have KMS Key Policies associated to each key and we restrict the key policies as much as possible to limit overly permissive users/roles.
The IAM Role that will be able to disable/delete the KMS Keys created by data.all stacks is one created by CDK of naming convention arn:aws:iam::ACCOUNT_ID:role/cdk-hnb659fds-cfn-exec-role-ACCOUNT_ID-REGION
The easiest way to delete KMS Keys would be to assume into this role and delete keys programmatically or via console once assumed into this role (NOTE: you may have to edit IAM trust relationships of this role to append your IAM entity to assume this role credentials)
Context: Version: 2.0.0 When we create DataSets or Environments, - S3, SQS, and KMS Keys get created. Upon deleting DataSets or Environments, the associated S3 Buckets do not get deleted, and neither do the SQS Queues or KMS Keys. We have been deleting S3 Buckets and SQS Queues manually from the console. However, deleting KMS Keys is not possible as the owner of the Keys is a specific role that creates them.
Requirement: We want to delete KMS Keys. We are looking for the easiest way to delete KMS Keys. We are particularly interested in cleaning up S3, SQS, and KMS on development or POC environments as we frequently create and delete a lot of datasets and environments.
Questions:
Thank you in advance for your assistance.