Unable to skip CA certificate in TLS config

[MarnixBouhuis]

Hey, we are deploying a mongodb cluster that needs to be reachable from outside the cluster over TLS. For this we want to use certificates provisioned using ACME / cert-bot and LetsEncrypt.

When configuring TLS we get the error saying:

TLS field requires a reference to the CA certificate which signed the server certificates. Neither secret (field caCertificateSecretRef) not configMap (field CaConfigMap) reference present

This error was introduced in this PR: https://github.com/mongodb/mongodb-kubernetes-operator/pull/1119

According to this issue the caConfigMap and caCertificateSecret should both be optional: https://github.com/mongodb/mongodb-kubernetes-operator/issues/519

Since we are using ACME / LetsEncrypt we do not have the CA certificate in a secret.

[phajduk]

@MarnixBouhuis I can confirm we have the same problem. I will comment more: MongoDB Kubernetes Operator suggests using cert-manager.io to manage certs. As described in their tutorial it's doable to generate LetsEncrypt certificate key using cert-manager. Without fixing the problem @MarnixBouhuis reported we can't easily integrate MongoDB with LetsEncrypt issued certificates. @dan-mckean any ideas how we should prioritize this? Mentioned PR addressed issue https://github.com/mongodb/mongodb-kubernetes-operator/issues/1114 and https://github.com/mongodb/mongodb-kubernetes-operator/issues/1054 but making CA file mandatory is not the proper fix if certificate is signed by trusted authority as LetsEncrypt. cc: @adamliesko @slaskawi

[fealebenpae]

Using Let's Encrypt-issued certificates with a database cluster deployed by the MongoDB Community Operator won't work, unfortunately, even if you were to work around the requirement for a CA certificate.

The issue is that the TLS certificate is validated both by clients connecting from outside the Kubernetes cluster using the replica set horizon hostnames defined in the MongoDBCommunity resource, and by clients within the Kubernetes cluster (including other mongod replica set members talking to each other) which use the Kubernetes cluster DNS to resolve the mongod pods. That's why in the cert-manager example the Certificate resource lists both the external DNS names but also the in-cluster DNS names that resolve to the individual mongod pods within the database cluster. Let's Encrypt will refuse to issue a certificate for the .svc.cluster.local domain and if you attempt to use a certificate with only the external domains then intra-cluster communication will break down.

[phajduk]

@fealebenpae you're right. This is why this ticket is a bit more wide. https://github.com/mongodb/mongodb-kubernetes-operator/blob/2cf2cae57576f8c62e4e6df2306df33353c7b295/controllers/replica_set_controller.go#L760 this function should allow to return non cluster.local domain.

[fealebenpae]

It's possible to override this with the MongoDB Enterprise operator by setting the spec.externalAccess.externalDoman field on the MongoDB resource. I suppose we can consider backporting this behavior to the Community operator.

[phajduk]

Do we have any timeline for this? @fealebenpae @dan-mckean @slaskawi