[Feature] - Ability to inject a CA cert into a cert-manager managed secret resource

[hawksight]

After reading through this cert-manager issue: https://github.com/cert-manager/cert-manager/issues/1571 I had a temporary bright spark, although the actual solution might not be the one suggested here.

tl;dr

A single secret where cert-manager controlls tls.crt and tls.key, but ca.crt is injected from trust-manager.

Problems

1) The pattern of a single secret containing tls.crt, tls.key and ca.crt seems to have become a standard for many tools mounting certificates in kubernetes. 2) There is an inconsistency that cert-manager issuers don't always provide a ca.crt and the current position is that this is no a pattern we want to promote. 3) A CA might not pass back a CA.crt, and the CA.crt might not be what the user actually wants anyway. 4) Using both cert-manager and trust-manager with a Secret + ConfigMap might sounds like a complete solution, but refer back to no1 where any number of k8s operators might not only work with 1 secret with defined keys. (redis operator, opensearch etc..)

Goals / Themes

1) We want user to be able to adopt cert-manager and trust-manager broadly across their k8s applications 2) We want to maintain certificate issuance and trust distribution via separate processes. 3) Ease of use and automation, eg. avoid having to manually or have custom scripts to combine secrets, rename fields etc..

Solution(s)

The main idea that struck me was off the back of this comment, specifcially:

Or even just a flag so we can manually set the ca.crt contents and cert-manager will include this in the secret as ca.crt

This seems a valid solution, could you manually insert the ca.crt into the secret itself. But why do this manually, trust-manager is about automating trust distribution. So what if trust-manager had a watch / trigger for Secrets that required a certain Bundle output to be inserted.

Just an idea but what about:

kind: Secret
metadata:
  annotations:
    trust.cert-manager.io/ca-inject: "true"
    trust.cert-manager.io/bundle: "my-bundle"
    cert-manager.io/alt-names: alpha.dev.peter-fiddes-gcp.jetstacker.net
    cert-manager.io/certificate-name: alpha-acme
    ...

Annotations are terrible? and there is probably a better way. But with ManagedFields it's now possible to safely have multiple controllers manage a resource (or so I believe). This could potentially be added in stages to the two controllers. For example if trust-manager was to watch secrets with valid annotations, then these annotations could be added using certificate.spec.secretTemplate.annotations.. eg..

spec:
  secretTemplate:
    annotations:
      trust.cert-manager.io/ca-inject: "true"
      trust.cert-manager.io/bundle: "my-bundle"

Basically no new cert-manager certificate CRD fields to link to bundle.. but perhaps in the future:

spec:
  caBundleRef: "my-bundle"

Assuming it was as above, you could potentially avoid annotations and have cert-manager ask trust-manager for the bundle ca at issuance time. Or alternatively, just mark the Secret field ca.crt in some fashion for trust-manager to do the injection.

Other solutions

• Utilise another controller / tool to bring the two parts together, similar to how secrets-transform currently manages extra keys. "pki-combinator"? :joy:

Concerns

• CA certs are not secret and probably should be in Secrets, but then again, a workload certificate tls.crt arguably doesn't need to be secret either, only the key.
• It's a completely new distribution mechanism for trust-manager, adding a new resource to watch & keep updated.
• What if the ca.crt changes from the bundle being propagated, would you want cert-manager to reissue the actual cert as well? I think this is combated with good practice.

Other

I haven't fully thought this out, just wanted to get a brain dump out and collaboratively find the best solution. Also see if this is a good idea, so please vote / comment with input.

[SpectralHiss]

I like this idea, one could even go 1 step further in convenience and annotate an Issuer so that all Certificates by that issuer get their ca.crt injected by trust-manager :thinking:

[wallrj]

The pattern of a single secret containing tls.crt, tls.key and ca.crt seems to have become a standard for many tools mounting certificates in kubernetes. There is an inconsistency that cert-manager issuers don't always provide a ca.crt and the current position is that this is no a pattern we want to promote. A CA might not pass back a CA.crt, and the CA.crt might not be what the user actually wants anyway. Using both cert-manager and trust-manager with a Secret + ConfigMap might sounds like a complete solution, but refer back to no1 where any number of k8s operators might not only work with 1 secret with defined keys. (redis operator, opensearch etc..)

What exactly does Redis Operator use the ca.crt file for?

For example, does a Redis node use ca.crt to verify the serving certificate of peers that it connects to? I suppose that makes sense if all node Pods are mounting the same Secret and using the same wildcard serving certificate. Or if each node Pod has a dedicated Secret with a dedicated DNS name for its serving certificate, but that all serving certificates are assumed to be signed by the same CA. This can already be achieved with CA issuer (and Vault and Venafi issuer I think). The issue you linked to (https://github.com/cert-manager/cert-manager/issues/1571) is about the ACME issuer, but does anyone use that for such peer certificates? Surely not Let's Encrypt + ACME, because the internal cluster DNS names of the peers couldn't be validated by Let's Encrypt.

I'd just be very interested to know exactly how the users are attempting to use the ca.crt file, because I suspect some of them may be shooting themselves in the foot.

I've also asked for details of the OpenLDAP case.

[michal-rybinski]

please also see https://github.com/cert-manager/cert-manager/issues/1571#issuecomment-1808305627 for another scenario where ca.crt is required by Istio and used for mTLS validation.

[Siegfriedk]

@wallrj i'm slighlyt surprised to see this pattern in the wild too. OpenSearch has the same requirement.

My current assumption is that enterprise is pushing for mTLS. Alternativly perhaps people have been burned to often with outdated trust stores on os level, tool level (java, ...) etc.?