feat: migrate to cert-manager

[f1ames]

This PR...

Changes

-

Fixes

-

Checklist

• [x] tested locally
• [ ] added new dependencies
• [ ] updated the docs
• [ ] added a test

[f1ames]

After working with cert-manager a bit and looking at other options, those are my conclusions and direction I suggest we should go. Starting from cert-manager:

Cert-manager

One of the most popular solutions for cert management in k8s space from what I see. https://cert-manager.io/.

Pros

• No configuration needed to deploy basic version. It will work out of the box.
• It's quite well-known solution when it comes to cert management in k8s.
• If there is already cert-manager deployed in the cluster it's easy to integrate it with monokle-admission-controller (we should enable it through helm charts).
• Cluster admin has full control over certificates, their configuration and how those are created.

Cons

• Seems like cert-manager is quite complex so it adds complexity to this solution as a whole. It might seem a bit too much for just "managing single certificate for admission webhook server".
• I can imagine someone is reluctant to deploy it to their cluster (due to complexity or resources restrains).
• Not the easiest to debug if something goes wrong.
• Default config is Self-Signed CA (considered not the most secure - however not sure if it matters that much for internal cluster communication 🤔).

How others do it?

Kyverno

I also looked at Kyverno and how they handle this. It has it's own cert-renewer. It uses Self-Signed CAs and also allows providing manually generated certificates.

For default configuration, it's also out of the box one - https://kyverno.io/docs/installation/customization/#default-certificates:

By default, Kyverno will automatically generate self-signed Certificate Authority (CA) and a leaf certificates for use in its webhook registrations. The CA certificate expires after one year. When Kyverno manage its own certificates, it will gracefully handle regeneration upon expiry.

This is done via dedicated custom made renewer util.

The other option is providing own certificat - https://kyverno.io/docs/installation/customization/#custom-certificates:

You can install your own CA-signed certificate, or generate a self-signed CA and use it to sign certificates for admission and cleanup controllers. Once you have a CA and X.509 certificate-key pairs, you can install these as Kubernetes Secrets in your cluster. If Kyverno finds these Secrets, it uses them, otherwise it will fall back on the default certificate management method. When you bring your own certificates, it is your responsibility to manage the regeneration/rotation process. Only RSA is supported for the CA and leaf certificates.

Kong

Did a bit of searching for how other tools handle/advise, on supplying certs for validation controllers. Found Kong (API platform on top of K8s AFAIU). They have a bit similar approach as Kyverno. Either one can provide own cert or, by default, the certificate is generated by install script or helm chart (so there is no built-in cert manager/renewer) - https://docs.konghq.com/kubernetes-ingress-controller/latest/deployment/admission-webhook/#main.

---

Conclusions

Based on the above I think cert-manager is viable options as it is well-known, allows easy setup and a lot of flexibility. However, we should provide simpler alternative. At this stage, using self signed certificate, generated by the install script during admission installation (e.g. as Kong does, through Helm) would be reasonable approach - it is simple and lightweight.

As for building dedicated, lightweight cert-renewer which will run in-cluster, maybe it can be done in later stages if needed. At this point I think it's not necessary.

WDYT @WitoDelnat @murarustefaan?

[murarustefaan]

To be honest, while I consider cert-manager a very useful approach for securing endpoints in the Kubernetes space, I would refrain from using it in this exact same context as it is a kubernetes plugin and not something you can count on being installed in all clusters.

Now, while installing cert-manager is pretty simple and straigtforward and you can even add it as dependency to a helm-chart, I believe would not make sense just for generating a self-signed certificate as it would require checking whether a existing installation is already in place to not overwrite the CRDs, add at least one permanently running pod in the cluster, creating a Issuer resource configured for self-signed CA's, generating and updating the certificate Resources, etc.

A different approach, depending on how resource-intensive this admission controller pod will be and whether we want it to scale horizontally, would be using an initContainer to generate the CA, public & private keys in:

• considering a highly-available deployment, a kubernetes Secret
• considering a single-pod deployment, an emptyDir volume. The initContainer would generate a new certificate when expired or close to expiry, or in the case of using emptyDir, would generate a new certificate each time (as emptyDir is ephemeral).

Now considering the CA & Certificate generation is complete (even if we would use cert-manager or our own generation method), the controller pod would need to make sure the APIServer has the correct CABundle for the ValidatingWebhookConfiguration, therefore on each bootstrap would need to refresh the ValidatingWebhookConfiguration resource with the valid CA bundle.

@WitoDelnat @olensmar what's you take on this?

[f1ames]

@murarustefaan thanks for the suggestion (and explanation). Seems like even more straightforward and maintenance-free approach. I'm taking a look into that 👍

• considering a highly-available deployment, a kubernetes Secret

I think this would be the direction. Admission controller should work without any interruptions/downtime, since being unavailable means broken deployment of other (validated) resources (well, depends on failurePolicy config option).

[WitoDelnat]

I believe we should refrain from cert-manager in the long run. I thought maybe it was easy to get started with and we could have something temporarily, but its indeed bringing a bulldozer when a hammer would be sufficient. Kyverno has a custom component to deal with certificate renewal.

There are plenty of strategies for generating (or providing custom) certificates. Helm Hook, initContainer, within the NodeJs process, cert-manager, etc. I think we should simply use a Kubernetes Secret as an indirection so this can change over time. It also provides custom certificates out of the box - you'd basically simply need to make sure its there before you bootstrap Monokle Admission Controller.

As for our first strategy, I'm not so opinionated and would go for the simplest solution. This is also why I made no remark against cert-managers as I expected more & better strategies to follow. If we don't think cert-manager is the simplest solution, then we can go for another. initContainers and Helm Hooks are both basically container jobs, I'd maybe lean to a Helm Hook as it works for both single-instance & high available setups.

[murarustefaan]

Yeah, indeed, but using helm hooks would regenerate certificates only during helm upgrades / installs, therefore would require the usage of certificates with large expirations (eg. at least over an year). An initContainer would solve this as pods are pretty ephemeral and even in an ideal cluster where nodes would not get shuffled often I highly doubt any pod would last more than say 3 months.

Furthermore, using helm hooks ties us to only shipping the admission controller setup using Helm, which I would refrein from doing in the long run; that's why i suggested using the bootstrapping of an initContainer as that's more easier to build on top in the future.

However, @f1ames, whichever way you go, pay attention to creating a ServiceAccount, Role & RoleBinding accordingly without granting more privileges than actually needed for reading, creating and updating the CA Secret & the ValidatingWebhookConfiguration resources.

[f1ames]

Closing in favor of https://github.com/kubeshop/monokle-admission-controller/pull/7.