Cert generated by init container is not removed during uninstall

[f1ames]

There is an init container which generates certificate used by webhook server (see here). And since the cert is generated as part of initialization logic and not helm chart / install script itself it is not removed on helm install / kubectl delete.

~It doesn't do much harm because the cert will be reused (or renewed when expired) when admission controller is again deployed to the same namespace. Still it will be better to have everything cleaned up.~ EDIT: see https://github.com/kubeshop/monokle-admission-controller/issues/23#issuecomment-1866423147 below.

For helm we can use helm hooks (like post-delete) to fix this. For install manifest (or as more generic approach) we can maybe use container-lifecycle-hooks (see here too).

This is how leftover cert looks like:

NAME                                      TYPE                                  DATA   AGE
secret/monokle-admission-controller-tls   kubernetes.io/tls                     2      2m12s

[f1ames]

It seems there is in fact negative impact there. Reinstalling to the same namespace, it seems to cause an issue with next deployment using previous cert and communication to webhook server is broken with:

Error from server (InternalError): Internal error occurred: failed calling webhook "monokle-admission-controller-server.monokle.svc": failed to call webhook: Post "https://monokle-admission-controller-server.monokle.svc:443/validate?timeout=10s": x509: certificate signed by unknown authority

I think this happens because there are two parts here:

• Server cert resource is created under secret/monokle-admission-controller-tls.
• Authority cert (with which server cert is signed) is added as webhook resource webhookConfig.clientConfig.caBundle.

And while server cert is reused, webhook caBundle is not propagated with authority cert on reinstall (because this cert is not there basically).

https://github.com/kubeshop/monokle-admission-controller/blob/52bddcca411fae0cc419a872fb5360c44402e044/admission-controller/init/src/index.ts#L50-L65

[f1ames]

There are two parts here:

• Making sure that uninstalling and installing again works (this is important).
• Improving certs cleanup after uninstalling.

Those are related and can be fixed at once (by single solution), but it depends on the approach. The important part is fixing the issue with reinstalling.