AndiDog
commented
1 year ago I looked at the stuck-Deleting clusters. See also chat where this was raised once more today. Each case seems different, so please have a look at the separate stories below.
Version hint
λ k get pod -n giantswarm capa-controller-manager-7d965c9b6f-zlgx2 -o yaml
[...]
image: docker.io/giantswarm/cluster-api-aws-controller:v1.5.2-gs-1ce7bb92Marcus commented that this image is manually built as it includes a fix we're waiting to get included upstream (https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/3871).
Andreas / grizzly / test2
It was stuck like this:
λ k tree -n org-andreas clusters.cluster.x-k8s.io "${WC}"
NAMESPACE NAME READY REASON AGE
org-andreas Cluster/test2 True 3d1h
org-andreas ├─AWSCluster/test2 False Deleting 3d1h
org-andreas ├─KubeadmControlPlane/test2 True 3d1h
org-andreas │ ├─AWSMachine/test2-control-plane-e88c76d5-9gq4z True 3d1h
org-andreas │ ├─AWSMachine/test2-control-plane-e88c76d5-kzpnl True 3d1h
org-andreas │ ├─AWSMachine/test2-control-plane-e88c76d5-ttzd2 True 3d1hErrors:
λ k describe awsc -n org-andreas test2 | tail -n 7
Type Reason Age From Message
---- ------ ---- ---- -------
Normal IRSA 3m5s (x13772 over 5h48m) irsa-capa-controller IRSA bootstrap deleted
Warning FailedDeleteSecurityGroup 62s (x279733 over 42h) aws-controller (combined from similar events): Failed to delete cluster managed SecurityGroup "sg-0dea1d842d96d4b1e": DependencyViolation: resource sg-0dea1d842d96d4b1e has a dependent object
status code: 400, request id: 791976eb-d4a9-4525-a40e-e15c5b035baf
Warning FailedDeleteSecurityGroup 62s (x279733 over 42h) aws-controller (combined from similar events): Failed to delete cluster managed SecurityGroup "sg-0dea1d842d96d4b1e": DependencyViolation: resource sg-0dea1d842d96d4b1e has a dependent object
status code: 400, request id: 791976eb-d4a9-4525-a40e-e15c5b035bafThe control plane was still running. Those 3 control plane EC2 instances still had the problematic security group test2-lb attached, among others (test2-controlplane and test2-node).
Trying to delete the KubeadmControlPlane object was hanging and not progressing. So I went for manual cleanup: detached the SGs from the 3 EC2 instances. CAPA was then able to delete the SGs. For the other CAPA resources such as AWSMachine and others, I had to delete manually. At the end, I had to restart the capi-kubeadm-control-plane-controller-manager pod because it would otherwise not get called in order to reconcile deletion of KubeadmControlPlane.
Wondering whether this happened to me because I deleted the cluster essentially like kubectl gs template cluster --provider capa […] | kubectl delete -f -. That’s only a guess. We should investigate if this is reproducible. Deleting in that way should work IMO.
Berk / grizzly / berk1
Stuck like this:
λ k tree -n org-giantswarm clusters.cluster.x-k8s.io berk1
NAMESPACE NAME READY REASON AGE
org-giantswarm Cluster/berk1 False Deleted 6d5h
org-giantswarm ├─AWSCluster/berk1 False Deleted 6d5h
org-giantswarm └─BackgroundScanReport/b8010aec-5d53-4b23-8418-3638f4a6c706 - 6dλ k describe -n org-giantswarm AWSCluster/berk1 | grep -A1 Finalizers:
Finalizers:
network-topology.finalizers.giantswarm.ioSo clearly it’s from our own finalizer.
Side note: capa-controller-manager logs are totally noisy because of this situation. The controller tries to reconcile deletion several times per second.
Our operator fails:
aws-network-topology-operator 2022-12-12T14:08:25.382276111+01:00 1.6708505053821046e+09 ERROR Reconciler error {"controller": "cluster", "controllerGroup": "cluster.x-k8s.io", "controllerKind": "Cluster", "cluster": {"name":"berk1","namespace":"org-giantswarm"}, "namespace": "org-giantswarm", "name": "berk1", "reconcileID": "2ce541ce-393e-4a3d-b5eb-e4ce6aa1dc8a", "error": "operation error EC2: DescribeTransitGatewayVpcAttachments, https response error StatusCode: 400, RequestID: f9ac87d6-fa9b-4471-9779-ed31d4a6309b, api error MissingParameter: Missing required parameter in request: Values of filter transit-gateway-id may not be empty."}And that seems because the transit gateway ID – as needed by the deletion code in func (r *TransitGateway) Unregister – isn't stored as annotation network-topology.giantswarm.io/transit-gateway:
λ k get -n org-giantswarm Cluster/berk1 -o jsonpath='{.metadata.annotations}' | jq
{
"cluster.giantswarm.io/description": "test",
"meta.helm.sh/release-name": "berk1",
"meta.helm.sh/release-namespace": "org-giantswarm",
"network-topology.giantswarm.io/mode": "GiantSwarmManaged"
}Maybe @bdehri has an idea what happened, based on how this cluster was created.
To me, it looks like there's no Transit Gateway attached to grizzly's VPC at all (someone else please confirm). Therefore, the controller would fail with The Management Cluster doesn't have a Transit Gateway ID specified at the time the cluster was created. Since logs are rotated out, I cannot confirm that. I left the cluster as-is since we're unclear about any leftovers at the moment.
AverageMarcus
tuladhar
Issue
New clusters are stuck in deleting on
grizzly.The problem needs to be confirmed, because it might be happening only for old clusters.