Trust part 2 - How to use a bundle?

[hawksight]

Now that I have installed trust, configured a bundle and I have a configMap propagated around the various namespaces of my cluster, I now need to feasibly use said CA bundle with applications. I think there are some usability and security concerns to accommodate here:

• We probably need some docs on https://cert-manager.io/docs/projects/trust/ on how best to consume said bundles.
• Most containers don't run as root and therefore won't be able to use update-ca-certificates at runtime
• Application might not have TLS configuration options.
• Deployment resources need updated mounting the relevant configmap
• Potentially creating a dependency on that configMap in the application life cycle.

Am I right in thinking that the best approach to update CA's trusted at runtime of a non-root container, is to effectively use a privileged init container with a shared volume with the application and some entrypoint.sh type magic? Something like the suggestion here

Perhaps it's easier to explain where I think I would like to be. I manage clusters with many tenants. I would like at runtime to automatically inject my trusted CA bundle to applications. Application use only that bundle without having to be specifically referenced by the tenants applications and deployments.

End result is that you effectively control trust at runtime through the cluster without requiring tenant changes. There is then potential to roll trust by refreshing or creating new bundles. This sort of reflects my experience that TLS is usually a platform / security concern and that application teams tend not to care about the runtime environment so much. Say for example an environment relying on internal CAs.

I'm not sure how achievable the above is, but perhaps some documentation on the trust project on typical use cases might be a start at least? At least highlight the gaps or limitations to address.

[larssb]

Look at this project > https://github.com/smallstep/autocert < for inspiration. We're using it currently for injecting the pub. CA cert of the CA we're self-hosting. It's working pretty well. However, sometimes, the only caveat, is that at Pod restart or re-schedule Pod workloads using the autocert project needs to be restarted twice before the autocert init type container kicks in and does its thing.

[SgtCoDFish]

Am I right in thinking that the best approach to update CA's trusted at runtime of a non-root container, is to effectively use a privileged init container with a shared volume with the application and some entrypoint.sh type magic? Something like the suggestion here

That's definitely an approach that can be taken! If we can rely on users to add init containers and update entrypoints that makes it a lot easier.

Part of the issue here is working out what's reasonable for users to do to be able to use the bundles we generate. Automating any of this is hard for several reason. I'll use "TMBundle" to mean "a bundle created by trust-manager".

1) Users may have different opinions on whether they want TMBundles to replace the container's trust store or to add to it. I think they should replace where possible but it's easy to imagine some will want to add. 2) We can't necc. rely on people running anything before their application (i.e. changing their entrypoint / adding an init container) 3) We can't easily tell what distro people are running inside a given container, and each distro may have a different path at which it writes its own trust store 4) We can't change application code which looks in a specific place for a trust store 5) I'm sure there are more!

I think the long term is probably that there has to be some kind of upstream k8s standard for this. Without that, I think automating this is always going to miss some edge cases.

Documenting and improving usage is definitely a priority of mine but it's prioritised behind public trust bundles and bundle output formats!

[evankanderson]

Possibly related KEP for "Trust Anchor Sets": https://github.com/kubernetes/enhancements/issues/3257

Some additional wrinkles:

1. Some organizations may also build custom CA certs into their base images
2. OpenSSL has a particular configuration with 7-letter (IIRC) symlinks to the actual cert files based on hashes of some of the CA data. I suspect these are scanned once when OpenSSL init is called, but I could be mistaken. For this reason, SSL_CERT_FILE may be easier to manage than SSL_CERT_DIR, though it may be possible to create non-symlink shortnames that will work in the trust ConfigMap.
3. Golang does not need the aforementioned symlinks.
4. Java uses a different Trust Store mechanism; it seems to recommend using a java-specific tool (keytool) to add certs to the Java trust store.
5. I don't know enough about Rust, but I suspect the situation is similar to Golang.

[gclawes]

Not only does java use a different Trust Store mechanism (which is sometimes tied to the distro's trust store), but different distros use different CA trust store configurations and different update commands:

• update-ca-certificates in Alpine and Debian-derived distros
• update-ca-trust on RedHat-derived distros

[SgtCoDFish]

Thanks @evankanderson and @gclawes! I think we're aware of the points you both raised, which is great validation! JKS files for Java is definitely on the roadmap.

I don't think any of the issues raised should be serious problems. The openssl symlink stuff would be wild to try and solve but I'm not aware of a need for us to actually solve that.

[hawksight]

Oh I really forgot that I opened this issue. Having a quick scan through there's a couple things that might help with the issues presented in comments:

• trust-manager now supports JKS output as well: https://github.com/cert-manager/trust-manager/pull/122. Since https://github.com/cert-manager/trust-manager/releases/tag/v0.5.0
• @SgtCoDFish wrote a blog that might be of use: https://venafi.com/blog/trust-manager/
• We also have this tutorial on cert-manager that I wrote about how you can use the bundle output.

I think particularly the last article (i wrote) should help address some of the comments about how to actually use this in Kubernetes with applications. It'll be different for certain languages of course and I don't cover all those previously mentioned, but it might be a good starting place.

I'm tempted to close out this issue and if there are specific challenges that haven't been addressed, they are raised as new concerns.

[erikgb]

I can add that we will also support password-less PKCS12 truststores in our next release, ref. https://github.com/cert-manager/trust-manager/pull/163 (with a few follow-up fix-PRs). 😉

[hawksight]

To update this thread, trust-manager has changed a fair bit since raising this issue, but the question does still remain on how best to consume a Bundle or the output of said resource.

We discussed this topic (thanks to @erikgb for bringing it up) on the cert-manager bi-weekly call yesterday (30th Nov 2023).

I created two action for myself to follow up:

• [ ] Review the trust-manager tutorial with the latest trust-manager release
• [ ] Initiate some contact with upstream on ClusterTrustBundle usage.

We also briefly mentioned some related issues / feature requests that might make Bundles more useful:

• https://github.com/cert-manager/trust-manager/issues/58
• https://github.com/cert-manager/trust-manager/issues/72

Clearly on big topic we identified which has many possible options is auto-mounting a bundle into the default locations. But default locations change per image base. @SpectralHiss has kindly offered to gather a list of the commone places used by various language and OS bases as a starting point in this effort.

The other area was making use of Bundle (or bundle output) in CRDs.

Both options are currently difficult or at the very least need something extra, or some documentation, so we will keep this issue open and hope to come to a clearer strategy for "how you use a Bundle".

[lknite]

I have found it common that a cm and secret are both too small to store the needed certs plus additional certs requiring a pvc to be needed instead.

If we expect folks to use an initContainer then why have something in every namespace? We could just use curl to get the latest ca-bundle using a trust-manager rest api.

Along that line though, if using a pvc, could we not use a ReadOnlyMany pvc so we only need one? Or possibly create a pvc in every namespace... or use a namespace label to indicate a ca-bundle pvc should be created there?

[lknite]

I have found it common that a cm and secret are both too small to store the needed certs plus additional certs requiring a pvc to be needed instead.

If we expect folks to use an initContainer then why have something in every namespace? We could just use curl to get the latest ca-bundle using a trust-manager rest api.

Along that line though, if using a pvc, could we not use a ReadOnlyMany pvc so we only need one?

... Perhaps we could create a stand-alone cli which, when given a docker image or oci reference, could go out and look up the image, performing checks to determine how the ca-bundle should be mounted. Then, later, trust-bundle can use that utility on the fly, with appropriate permissions granted via namespace labels to mount the ca-bundle to the correct location. ... if the k8s clusters are airgapped we can use the cli to get the technique needed and store that in a cm, perhaps the tool could output a configmap or label the namespace, and that could be used by trust-manager to add the ca-bundle on the fly.

[cert-manager-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. /lifecycle stale

[hawksight]

/remove-lifecycle stale