Webhook configurations are not cleaned up after tenant deletion

SOF3 commented 2 years ago

What happened?

After tenant deletion, the ValidatingWebhookConfiguration remains present.

What did you expect to happen?

The ValidatingWebhookConfiguration should be deleted since the client no longer exists. Otherwise, this may cause problems when a tenant with the same name is created again.

How can we reproduce it (as minimally and precisely as possible)?

Following manually-setup.md up to kubectl apply -f sample_tenant.yaml.

$ kubectl --context zoo apply -f another.yaml
tenant.tenant.kubezoo.io/111111 created

$ kubectl --context zoo get tenant 111111 -o jsonpath='{.metadata.annotations.kubezoo\.io\/tenant\.kubeconfig\.base64}' | base64 --decode > 111111.kubeconfig

$ kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io
No resources found

$ kubectl --kubeconfig 111111.kubeconfig apply -f validating-hook.yml
validatingwebhookconfiguration.admissionregistration.k8s.io/webhook.example.com created

$ kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io
NAME                         WEBHOOKS   AGE
111111-webhook.example.com   4          19s

$ kubectl get ns
NAME                     STATUS   AGE
111111-default           Active   2m32s
111111-kube-node-lease   Active   2m32s
111111-kube-public       Active   2m32s
111111-kube-system       Active   2m32s
default                  Active   14m
kube-node-lease          Active   14m
kube-public              Active   14m
kube-system              Active   14m
local-path-storage       Active   14m

$ kubectl --context zoo delete tenant 111111
tenant.tenant.kubezoo.io "111111" deleted

$ kubectl get ns
NAME                 STATUS   AGE
default              Active   14m
kube-node-lease      Active   14m
kube-public          Active   14m
kube-system          Active   14m
local-path-storage   Active   14m

$ kubectl get validatingwebhookconfigurations.admissionregistration.k8s.io
NAME                         WEBHOOKS   AGE
111111-webhook.example.com   4          46s

Software version

docker.io/kubezoo/kubezoo:v0.1.0

caohe commented 2 years ago

Thanks for ur feedback! The garbage collection mechanism for tenant's cluster-scope resources is not perfect. We will improve this and please feel free to contribute it.

caohe commented 1 year ago

To address this, the tenant controller needs to delete all kinds of cluster-scoped resources when a tenant is deleted.

Considering there are 20+ kinds of cluster-scoped resources, I think maybe we can delete some common ones:

PersistentVolume
StorageClass
PriorityClass
MutatingWebhookConfiguration
ValidatingWebhookConfiguration
CustomeResourceDefinition
Namespace (already supported)
ClusterRole (already supported)
ClusterRoleBinding (already supported)

To delete cluster-scoped resources, we can

list all objects of this kind
check whether an object belongs to this tenant
delete this object if it belongs to this tenant. In the case of CRDs, we can use cascading delete to cluster-scoped CRs at the same time.

WDYT? /cc @Silverglass

kubewharf / kubezoo