Handle certificate rotation

[flavio]

Description

Our controller uses two different CA roots:

1. the one generated by cert-manager, which generates the CRD webhook leaf certificate
2. The one generated by the PolicyServerController reconciliation loop the first time a policy server is created, which is used to generate the policy servers' TLS certificate.

There are a few problems with this approach:

1. it creates a dependency on cert-manager, see: kubewarden/kubewarden-controller#422
2. we don't rotate policy server certificates, therefore they will eventually expire.
3. both cert-manager and our implementation on the PolicyServer side do not rotate the CA root (see https://github.com/cert-manager/cert-manager/issues/2478)

Solution

We need to implement our certificate rotation logic inside the controller. Specifically, we need to:

• Generate a root CA and use it to configure the webhooks (CRD and policy validating/mutating webhooks)
• Generate the CRD webhook TLS certificate to configure the controller webhook server.
• Generate the TLS policy server certificates every time a new policy server is created
• Implement a CertController that periodically checks if the certificates are expired and regenerates them using a lookahead interval. If a TLS certificate is expired we can re-generate it, and it will be picked up by the servers eventually. If the CA root is expired we need to regenerate all the leaf certificates and implement the logic needed to prevent downtime during this phase.

Tasks

• [x] kubewarden/kubewarden-controller#820 https://github.com/kubewarden/helm-charts/pull/488
• [x] kubewarden/kubewarden-controller#819 https://github.com/kubewarden/kubewarden-controller/pull/829
• [x] kubewarden/kubewarden-controller#818
• [x] https://github.com/kubewarden/kubewarden-controller/issues/854
• [x] kubewarden/kubewarden-controller#861
• [x] kubewarden/kubewarden-controller#860
• [x] kubewarden/helm-charts#512
• [x] kubewarden/rfc#39
• [x] https://github.com/kubewarden/docs/issues/252
• [ ] https://github.com/rancher/kubewarden-ui/issues/839
• [ ] kubewarden/kubewarden-controller#864
• [x] kubewarden/kubewarden-controller#862
• [ ] kubewarden/helm-charts#516

[flavio]

Worth a look, something we could leverage: https://github.com/open-policy-agent/cert-controller

[kkaempf]

I always wonder why every project (like kucero for CaaSP) has to re-implement certificate handling. This should be built into Kubernetes.

[fabriziosestito]

Worth a look, something we could leverage: https://github.com/open-policy-agent/cert-controller

We decided not to leverage https://github.com/open-policy-agent/cert-controller since it does not support zero-downtime CA rotation. Also, we will need to fork it to adapt it to our use case, as we configure policy webhooks dynamically. Finally, it does not fully support leader election due to this bug.

[flavio]

Closing, all the mandatory tasks have been done.