`mutatingwebhookconfiguration`/`validatingwebhookconfiguration` objects left behind after application removal

ca-scribner commented 3 years ago

Sometimes (always?) when doing juju remove-application admission-webhook, mutatingwebhookconfiguration/validatingwebhookconfiguration objects are left on the cluster. This will block all pod creation because the webhooks will fail (as the services/pods they point to are removed). This can be seen in the kubernetes event logs as timeouts on calls to admission-webhook.kubeflow.org:

$ kubectl get events -n kubeflow
LAST SEEN   TYPE      REASON                  OBJECT                                                        MESSAGE
93s         Warning   FailedCreate            statefulset/kfp-api-operator                                  create Pod kfp-api-operator-0 in StatefulSet kfp-api-operator failed error: Internal error occurred: failed calling webhook "admission-webhook.kubeflow.org": Post "https://admission-webhook.kubeflow.svc:443/apply-poddefault?timeout=30s": Service Unavailable
92s         Warning   FailedCreate            statefulset/kubeflow-volumes-operator                         create Pod kubeflow-volumes-operator-0 in StatefulSet kubeflow-volumes-operator failed error: Internal error occurred: failed calling webhook "admission-webhook.kubeflow.org": Post "https://admission-webhook.kubeflow.svc:443/apply-poddefault?timeout=30s": Service Unavailable

Possible resolutions:

ensure juju tracks these objects (do they have the typical juju metadata needed to destroy with an application?)
handle removal of these in a charm remove hook
add parentage on these objects so if the primary admission-webhook objects go down so do these (although that could tear these down accidentally)

ca-scribner commented 3 years ago

Not sure if this happens always or if it just happens maybe during a juju destroy-model or a --force on application/model removal? May need to try different scenarios

DomFleischmann commented 2 years ago

This should be investigated a bit more, we need to know if it only happens with --force or always. Either way, if it occurs outside of --force, it seems like a juju bug that should be filed and linked in this issue.

ca-scribner commented 2 years ago

Closing this bug as currently juju correctly removes the mutating webhook for:

juju remove-application admission-webhook
juju remove-application admission-webhook --force Likely the cause has been fixed in Juju since this bug was filed

canonical / admission-webhook-operator

`mutatingwebhookconfiguration`/`validatingwebhookconfiguration` objects left behind after application removal #7