[EKS] [request]: Automatic clean-up of Kubernetes-provisioned AWS resources on cluster deletion

[valorl]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request Deleting a cluster should automatically clean-up any AWS resources that were provisioned by Kubernetes controllers.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? The most common scenario is when you have Kubernetes Services of Type=LoadBalancer, which results in LBs being provisioned. Once you delete the cluster, the LBs remain and have to be cleaned-up manually if you e.g. want to also destroy the subnets. This likely also applies to other third-party controllers/operators that manage external resources.

The below workarounds are not great, as they don't work well in an Infrastructure-as-Code managed environment. I think fixing this would give EKS better support for ephemeral clusters. Also, it appears that AKS is able to handle this.

Are you currently working around this issue? For load balancers, there are currently two ways of handling this:

• Delete all Services (of Type: LoadBalancer) and wait for resources to be cleaned up prior to EKS deletion
• Script / manual cleanup after cluster deletion

Additional context Prior discussions/reports: https://github.com/kubernetes/kubernetes/issues/93390

[1mamute]

This is happening a lot to me using Terraform. My solution at the moment is cleaning up every namespace before running terraform destroy.

If we can properly tag the resources created by the AWS Cloud Controller Manager, we can then clean them up at the DestroyCluster API call.

That solution would not only resolve this issue from IaC tools like Terraform but to all clusters, making sure that resources created by the cloud controller manager would then be cleaned up when the cluster is destroyed.

E.g.: I create a LoadBalancer service.
The AWS cloud controller manager creates a CLB and several network interfaces.
The cloud controller manager tags those resources with k8s.io/created-by: cloud-controller-manager and k8s.io/cluster-name: foo-cluster.
I want to destroy the cluster and call DestroyCluster (or terraform destroy).
The DestroyCluster API call searches for possible resources with these tags and destroys them before destroying the cluster itself.

[bobh66]

Also need to delete the EBS volumes that are provisioned by the CSI controller for PVCs, and the Security Group that was created for the EKS cluster.

[negz]

Breadcrumbs to https://github.com/aws/containers-roadmap/issues/1044, which I believe tracks something similar.