Closed kaykhancheckpoint closed 3 years ago
Hey @kaykhancheckpoint did you figure out?
I'm thinking of adding an Istio ServiceEntry for mongodb's clusterIP service and then exposing the ServiceEntry through Istio's Gateway+VirtualService... Not sure this will workout or even if it does, there must be an elegant way than this...
@navndn No i didn't. I've moved away from this operator for the time being, its still in pre-release and there are some important features that it doesn't have right now which i need.
For now im using mongodb helm chart owned by bitnami.
But, I'll keep an eye on this and see how it progresses, maybe ill try it again in the future once there is a stable release
@kaykhancheckpoint thanks for response. Even I decided to use bitnami chart for now
@kaykhancheckpoint thanks for your question
You can configure your mongodb deployment to be reachable from outside the kubernetes cluster with a LoadBalancer or NodePort (like with the bitnami helm chart). We don't provide examples as of now, as we do believe all connections should be over SSL and it's not straightforward to provide SSL encrypted connections from outside the cluster.
We do have a solution for external access to the cluster with SSL currently implemented with the enterprise operator and we will provide it soon to the community/open-source operator (this repo): it is based on split horizon, see point 8 here for how to configure in enterprise https://docs.mongodb.com/kubernetes-operator/master/tutorial/deploy-replica-set/.
@kaykhancheckpoint @navndn could you please give us a list of important features you need so that we can incorporate them to our roadmap? Either here or on https://feedback.mongodb.com/forums/924355-ops-tools.
FYI we plan to have all the authentication and encryption features are implemented (including split-horizon) within a month, you can follow the progress here: https://github.com/mongodb/mongodb-kubernetes-operator/projects/2.
Hi everyone, I have a question related to this issue.
I deployed mongodb community operator on Openshift. I can connect to my resources from outside via NodePort and LoadBalancer option but I can not connect it via routes (either secure or insecure).
when i try to connect it gives this error: "Error: network error while attempting to run command 'isMaster' on host '"
Is there a way to use route without ssl even though I know using ssl is the preferred way?
Thank you!
Hi @LouisPlisso , hope you are good.
I deployed mongodb community operator on Openshift. I can connect to my resources from outside via NodePort and LoadBalancer. But the problem is sometimes these service resources direct me to secondary mongodb resource instead of primary and I can not run any query on secondary resource because it gives the error below:
Is there a way to always direct connections to primary resource?
Thank you
Hi again @chatton , I would like to ask you my last question.
I can access my mongodb replica sets from outside cluster via NodePort or LoadBalancer but they sometimes direct connection to primary and sometimes to secondary. I would like to ask if there is a way to know which pod is exactly primary (I deleted the pods and they switched roles in some occassions) so that I won't face any NotMasterNoSlaveOk problem.
I learnt that I can send "mongo ... --eval rs.slaveOk=true ..." while connecting to use secondary without any denial but is there a way to set slaveOk=true from any config file. I searched internet people talked about editing "/etc/mongorc.js" file but our pods don't have that directory.
Could you please help me with this issue? I am stuck in this phase.
Thank you very much!
I was able to get this operator to work with some modifications, using the (undocumented) split horizon functionality available in MongoDB. This uses SNI, thus does require TLS to be enabled; if you have already have a PKI of sorts it's a lot easier.
In summary, besides the default Service
that the operator creates, I created 3 more Service
objects (type NodePort
) referencing the statefulset.kubernetes.io/pod-name
label, then created DNS records referencing their external IP addresses.
Then, configure the MongoDB
resource like so:
apiVersion: mongodb.com/v1
kind: MongoDB
metadata:
name: production
namespace: mongodb-operator
spec:
members: 3
replicaSetHorizons:
- horizon: production-0.mongodb-operator.example.com:31181
- horizon: production-1.mongodb-operator.example.com:31182
- horizon: production-2.mongodb-operator.example.com:31183
# ...
(the replicaSetHorizons
was added by my patch - it won't work with the current operator!)
Hi @yjwong ,
I am trying your fork for accessing externally. But I am having an error in stateful set where image name is set incorrectly.
Do you know how can we fix this problem? (btw 4.2.6 or 4.2.7 either definition in "...cr.yaml" not works)
This uses SNI, thus does require TLS to be enabled
Thanks @yjwong for the pointer. I almost spent a day for this.
Just sharing the steps below to help others and most likely my future self.
Generate the certificates with something like mkcert
. Refer this
Import them as configmap and secret, respectively.
30000,30001,30002 will be available on all nodes.
---
apiVersion: v1
kind: Service
metadata:
name: example-mongodb-instance-0
spec:
type: NodePort
ports:
- port: 27017
targetPort: 27017
nodePort: 30000
selector:
statefulset.kubernetes.io/pod-name: example-mongodb-0
---
apiVersion: v1
kind: Service
metadata:
name: example-mongodb-instance-1
spec:
type: NodePort
ports:
- port: 27017
targetPort: 27017
nodePort: 30001
selector:
statefulset.kubernetes.io/pod-name: example-mongodb-1
---
apiVersion: v1
kind: Service
metadata:
name: example-mongodb-instance-2
spec:
type: NodePort
ports:
- port: 27017
targetPort: 27017
nodePort: 30002
selector:
statefulset.kubernetes.io/pod-name: example-mongodb-2
rs.mongo.k3s is what we will use in connection string later Screenshot is taken from a private hosted zone
Include the spec replicaSetHorizons
. The key "horizon" here does not matter.
Also ensure certificateKeySecretRef
and caConfigMapRef
is using the ones that you have created.
spec:
members: 3
replicaSetHorizons:
- horizon: "nodes.mongo.k3s:30000"
- horizon: "nodes.mongo.k3s:30001"
- horizon: "nodes.mongo.k3s:30002"
type: ReplicaSet
version: "4.2.6"
security:
tls:
enabled: true
certificateKeySecretRef:
name: mongo-tls-cert # create
caConfigMapRef:
name: mongo-ca
authentication:
modes: ["SCRAM"]
mongo mongodb+srv://USER:PASS@rs.mongo.k3s/
` apiVersion: v1 kind: Service metadata: annotations: service.beta.kubernetes.io/azure-dns-label-name: example-mongodb-prod name: example-mongodb-prod-lb spec: loadBalancerIP: 111.111.111.111 type: LoadBalancer selector: statefulset.kubernetes.io/pod-name: example-mongodb-0 ports:
`
I did it with a cluster loadbalancer, i use azure kubernetes service, im able to access mongodb using assigned dns name or ip both from mongo shell & mongo compass
Hello @yjwong & @sagittaros Thank you for devoting the time to adding & testing the ReplicasetHorizons feature. I have upgraded to the latest version of the operator, however for some reason I still cannot reach the ReplicaSet from external sources:
/ # mongo "mongodb://public-ip-0:27017,public-ip-1:27017,public-ip-2:27017/?replicaSet=rs-name" --ssl --sslAllowInvalidCertificates
MongoDB shell version v4.0.5
connecting to: mongodb://public-ip-0:27017,public-ip-1:27017,public-ip-2:27017/?gssapiServiceName=mongodb&replicaSet=rs-name
2021-07-06T12:17:47.715+0000 I NETWORK [js] Starting new replica set monitor for rs-name/public-ip-0:27017,public-ip-1:27017,public-ip-2:27017
2021-07-06T12:17:47.729+0000 W NETWORK [ReplicaSetMonitor-TaskExecutor] SSL peer certificate validation failed: self signed certificate
2021-07-06T12:17:47.730+0000 W NETWORK [js] SSL peer certificate validation failed: self signed certificate
2021-07-06T12:17:47.733+0000 I NETWORK [ReplicaSetMonitor-TaskExecutor] Successfully connected to public-ip-0:27017 (1 connections now open to public-ip-0:27017 with a 5 second timeout)
2021-07-06T12:17:47.734+0000 I NETWORK [js] Successfully connected to public-ip-1:27017 (1 connections now open to public-ip-1:27017 with a 5 second timeout)
2021-07-06T12:17:47.812+0000 I NETWORK [js] changing hosts to rs-name/rs-name-0.rs-name-svc.mongodb.svc.cluster.local:27017,rs-name/rs-name-1.rs-name-svc.mongodb.svc.cluster.local:27017,rs-name/rs-name-2.rs-name-svc.mongodb.svc.cluster.local:27017:27017 from rs-name/public-ip-2:27017,public-ip-1:27017,public-ip-0:27017
2021-07-06T12:17:48.518+0000 W NETWORK [js] Unable to reach primary for set rs-name
I have the following configuration in place:
replicaSetHorizons:
- horizon: public-ip-0:27017
- horizon: public-ip-1:27017
- horizon: public-ip-2:27017
Note: Executing the mongo command with the same connection string on a pod residing on the cluster where the ReplicaSet is deployed results in successful connection. I created 3 services with LoadBalancer type to expose each ReplicaSet members (PRIM-SEC-SEC).
Is there any way to get rid of this error?
@dgdevops with a quick glance (and with a fuzzy memory), I think the issue lies in the usage of IP addresses where it should have been a list of domains.
External cluster access leverages split horizon, which is a DNS feature.
Hello @sagittaros, Thank you for getting back to me so quickly. I forgot to mention that my first attempt was to use domain names in the replicaSetHorizons settings but I stumbled upon the following error:
connecting to: mongodb://cg0.public-domain:27017,cg1.public-domain:27017,cg2.public-domain:27017/?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs-name
2021-07-07T17:32:05.122+0200 I NETWORK [js] Starting new replica set monitor for rs-name/cg0.public-domain:27017,cg1.public-domain:27017,cg2.public-domain:27017
2021-07-07T17:32:05.123+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to cg2.public-domain:27017
2021-07-07T17:32:05.123+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to cg1.public-domain:27017
2021-07-07T17:32:05.124+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to cg0.public-domain:27017
2021-07-07T17:32:05.262+0200 W NETWORK [ReplicaSetMonitor-TaskExecutor] SSL peer certificate validation failed: unable to get local issuer certificate
2021-07-07T17:32:05.318+0200 W NETWORK [ReplicaSetMonitor-TaskExecutor] SSL peer certificate validation failed: unable to get local issuer certificate
2021-07-07T17:32:05.347+0200 W NETWORK [ReplicaSetMonitor-TaskExecutor] SSL peer certificate validation failed: unable to get local issuer certificate
2021-07-07T17:32:05.350+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to rs-name-1.rs-name-svc.mongodb.svc.cluster.local:27017
2021-07-07T17:32:05.400+0200 I NETWORK [ReplicaSetMonitor-TaskExecutor] Confirmed replica set for rs-name is rs-name/rs-name-0.rs-name-svc.mongodb.svc.cluster.local:27017,rs-name-1.rs-name-svc.mongodb.svc.cluster.local:27017,rs-name-2.rs-name-svc.mongodb.svc.cluster.local:27017
2021-07-07T17:32:05.407+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to rs-name-0.rs-name-svc.mongodb.svc.cluster.local:27017
2021-07-07T17:32:05.410+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to rs-name-2.rs-name-svc.mongodb.svc.cluster.local:27017
2021-07-07T17:32:05.735+0200 W NETWORK [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs-name
2021-07-07T17:32:05.736+0200 I NETWORK [ReplicaSetMonitor-TaskExecutor] Cannot reach any nodes for set rs-name. Please check network connectivity and the status of the set. This has happened for 1 checks in a row.
Configuration used:
replicaSetHorizons:
- horizon: cg0.public-domain:27017
- horizon: cg1.public-domain:27017
- horizon: cg2.public-domain:27017
Connectivity test:
# mongo --host cg0.public-domain
MongoDB shell version v4.2.6
connecting to: mongodb://cg0.public-domain:27017/?compressors=disabled&gssapiServiceName=mongodb
MongoDB server version: 4.2.6
rs-name:SECONDARY>
# mongo --host cg1.public-domain
MongoDB shell version v4.2.6
connecting to: mongodb://cg1.public-domain:27017/?compressors=disabled&gssapiServiceName=mongodb
MongoDB server version: 4.2.6
rs-name:PRIMARY>
# mongo --host cg2.public-domain
MongoDB shell version v4.2.6
connecting to: mongodb://cg2.public-domain:27017/?compressors=disabled&gssapiServiceName=mongodb
MongoDB server version: 4.2.6
rs-name:SECONDARY>
@dgdevops I see some SSL error there, and I suspect you don't have both the internal domains (kubernetes svc) and public domains in your self-signed subjectAltName
field. Refer my certificate-ext.conf
below.
Ok, let me just share how I did this. Mostly strung together from memory and zsh history so it may not work 100%.
# pre-generated with mkcert
CA_CERT=ca.crt
CA_KEY=ca.pem
# To be generated
SERVER_PRIVATE_KEY=mongo-key.pem
SERVER_CERT=mongo-crt.pem
# generate server private key (RSA)
openssl genrsa -out ${SERVER_PRIVATE_KEY} 4096
# generate CSR and submitting certificate subject details
openssl req -new \
-config certificate.conf \
-key ${SERVER_PRIVATE_KEY} \
-out csr.pem
# SELF SIGNING HERE
openssl x509 -req \
-in csr.pem \
-CA ${CA_CERT} \
-CAkey ${CA_KEY} \
-CAcreateserial \
-out ${SERVER_CERT} \
-days 3650 \
-extfile certificate-ext.conf
echo "Your certificate is generated: ${SERVER_CERT}"
export KUBECONFIG=$HOME/.kube/config
kubectl create configmap mongo-ca --from-file=${CA_CERT}
kubectl create secret tls mongo-tls-cert --cert=${SERVER_CERT} --key=${SERVER_PRIVATE_KEY}
certificate-ext.conf
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.MYAPP-mongodb-svc.default.svc.cluster.local
DNS.2 = nodes.PUBLICDOMAIN.k3s
DNS.3 = rs.PUBLICDOMAIN.k3s
Hello @sagittaros , Thank you for getting back to me with an update. I added the public domains to the self-signed certificate as you suggested, the SSL related errors disappeared but the connection is still not possible as seen below:
# mongo "mongodb://cg0.public-domain:27017,cg1.public-domain:27017,cg2.public-domain:27017/?replicaSet=rs-name"
MongoDB shell version v4.2.6
connecting to: mongodb://cg0.public-domain:27017,cg1.public-domain:27017,cg2.public-domain:27017/?compressors=disabled&gssapiServiceName=mongodb&replicaSet=rs-name
2021-07-08T11:44:00.612+0200 I NETWORK [js] Starting new replica set monitor for rs-name/cg0.public-domain:27017,cg1.public-domain:27017,cg2.public-domain:27017
2021-07-08T11:44:00.613+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to cg1.public-domain:27017
2021-07-08T11:44:00.613+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to cg0.public-domain:27017
2021-07-08T11:44:00.614+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to cg2.public-domain:27017
2021-07-08T11:44:00.800+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to rs-name-0.rs-name-svc.mongodb.svc.cluster.local:27017
2021-07-08T11:44:00.857+0200 I NETWORK [ReplicaSetMonitor-TaskExecutor] Confirmed replica set for rs-name is rs-name/rs-name-0.rs-name-svc.mongodb.svc.cluster.local:27017,rs-name-1.rs-name-svc.mongodb.svc.cluster.local:27017,rs-name-2.rs-name-svc.mongodb.svc.cluster.local:27017
2021-07-08T11:44:00.860+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to rs-name-2.rs-name-svc.mongodb.svc.cluster.local:27017
2021-07-08T11:44:00.866+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to rs-name-1.rs-name-svc.mongodb.svc.cluster.local:27017
2021-07-08T11:44:01.287+0200 W NETWORK [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs-name
2021-07-08T11:44:01.288+0200 I NETWORK [ReplicaSetMonitor-TaskExecutor] Cannot reach any nodes for set rs-name. Please check network connectivity and the status of the set. This has happened for 1 checks in a row.
2021-07-08T11:44:01.763+0200 W NETWORK [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs-name
2021-07-08T11:44:01.764+0200 I NETWORK [ReplicaSetMonitor-TaskExecutor] Cannot reach any nodes for set rs-name. Please check network connectivity and the status of the set. This has happened for 2 checks in a row.
2021-07-08T11:44:02.114+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to rs-name-0.rs-name-svc.mongodb.svc.cluster.local:27017
2021-07-08T11:44:02.115+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to rs-name-1.rs-name-svc.mongodb.svc.cluster.local:27017
2021-07-08T11:44:02.117+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Connecting to rs-name-2.rs-name-svc.mongodb.svc.cluster.local:27017
2021-07-08T11:44:02.279+0200 W NETWORK [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs-name
2021-07-08T11:44:02.279+0200 I NETWORK [ReplicaSetMonitor-TaskExecutor] Cannot reach any nodes for set rs-name. Please check network connectivity and the status of the set. This has happened for 3 checks in a row.
2021-07-08T11:44:02.750+0200 W NETWORK [ReplicaSetMonitor-TaskExecutor] Unable to reach primary for set rs-name
2021-07-08T11:44:02.750+0200 I NETWORK [ReplicaSetMonitor-TaskExecutor] Cannot reach any nodes for set rs-name. Please check network connectivity and the status of the set. This has happened for 4 checks in a row.
^C2021-07-08T11:44:03.134+0200 I CONTROL [main] shutting down with code:0
Additionally I also tried connecting to the replicaset from Mongo Compass as well but got the following error:
getaddrinfo ENOTFOUND rs-name-0.rs-name-svc.mongodb.svc.cluster.local
The error above makes me think that for some reason the inter-cluster domain names are not translated to public domains so the mongo clients (Compass and mongo shell) want to connect to the primary using it's inter-cluster domain name which makes it unaccessible from outside of the cluster.
Mind sharing your deployment specs?
Sure @sagittaros, here it is:
apiVersion: mongodbcommunity.mongodb.com/v1
kind: MongoDBCommunity
metadata:
name: rs-name
namespace: mongodb
spec:
additionalMongodConfig:
net.ssl.allowInvalidCertificates: true
net.ssl.mode: preferTLS
storage.wiredTiger.engineConfig.journalCompressor: zlib
members: 3
persistent: true
replicaSetHorizons:
- horizon: cg0.public-domain:27017
- horizon: cg1.public-domain:27017
- horizon: cg2.public-domain:27017
security:
authentication:
modes:
- SCRAM
roles:
- db: admin
privileges:
- actions:
- listCollections
resource:
collection: ""
db: ""
role: customMonitoring
roles: []
tls:
caConfigMapRef:
name: mongodb-cg-rs-cert
certificateKeySecretRef:
name: mongodb-cg-rs-cert
enabled: true
statefulSet:
spec:
template:
spec:
containers:
- env:
- name: KUBE_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: KUBE_LABEL_hostname
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: CHECK_INTERVAL
value: "10"
image: <OMITTED>:202104081338
name: mongo-<OMITTED>
resources:
limits:
cpu: 200m
memory: 200M
requests:
cpu: 150m
memory: 150M
- args:
- --mongodb.uri=mongodb://admin:<OMITTED>@127.0.0.1:27017/admin?ssl=true&sslinsecure=true
image: eu.gcr.io/mgmt-238514/public/bitnami/mongodb-exporter:0.11.0
name: mongodb-exporter
resources:
limits:
cpu: 150m
memory: 150M
port: 9216
requests:
cpu: 100m
memory: 100M
imagePullSecrets:
- name: gcr-registry
initContainers:
- name: mongod-posthook
resources:
limits:
cpu: 200m
memory: 300M
requests:
cpu: 100m
memory: 200M
- name: mongodb-agent-readinessprobe
resources:
limits:
cpu: 200m
memory: 300M
requests:
cpu: 100m
memory: 200M
volumeClaimTemplates:
- metadata:
name: data-volume
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 30Gi
storageClassName: persistent-ssd
type: ReplicaSet
users:
- db: admin
name: admin
passwordSecretRef:
key: ""
name: mongodb-cg-admin-secret
roles:
- db: admin
name: clusterAdmin
- db: admin
name: userAdminAnyDatabase
- db: admin
name: readWriteAnyDatabase
scramCredentialsSecretName: mongodb-cg-admin
version: 4.2.6
I just tried the same connection string format on my machine.
mongo \
mongodb://USER:PASS@rs.DOMAIN:30000,rs.DOMAIN:30001,rs.DOMAIN:30002/?replicaSet=my-mongodb \
--tls
I have to add --tls
to make it work.
Also, I normally use mongo-shell, which is stricter in root CA check (I recall Compass behave more similarly to mongo-shell than mongo). You have to install your CA cert for Compass to work.
mongosh \
mongodb://USER:PASS@rs.DOMAIN:30000,rs.DOMAIN:30001,rs.DOMAIN:30002/?replicaSet=my-mongodb \
--tls \
--tlsCAFile=/home/ubuntu/.local/share/mkcert/rootCA.pem
Thanks @sagittaros for checking, I posted the deployment manifest above.
I added the --tls
flag, getting the same errors.
@dgdevops the deployment spec looks good to me except:
caConfigMapRef:
name: mongodb-cg-rs-cert
certificateKeySecretRef:
name: mongodb-cg-rs-cert
caConfigMapRef
should be a CA cert, not to be confused with the server/mongo certificate.
I also suggest you try connecting using mongosh:
mongosh \
mongodb://cg0.public-domain:27017,cg1.public-domain:27017,cg2.public-domain:27017/?replicaSet=rs-name \
--tls \
--tlsCAFile=/path/to/rootCA.pem
@sagittaros , I am using Kubernetes cert-manager to issue the certificates and the caConfigMapRef
pointing to mongodb-cg-rs-cert
is generated from the mongodb-cg-rs-cert
Kubernetes secret's ca.crt field that is created by Let's encrypt.
New doc added: docs/external_access.md
and samples in config/samples/external_access
Hello @sagittaros, Many thanks for the suggestions, eventually I made it work. @priyolahiri. thank you for the documentation.
Great! What was the issue? @dgdevops
Hey @sagittaros, I am not entirely sure, the only difference I can see between the working & not working scenarios is that I redeployed the replicaset with the ReplicaSetHorizons feature enabled from scratch. Previously I added the ReplicaSetHorizons parameter to the existing replicaset.
Hi i want to expose the mongodb publicly. But im not sure where to set mongodb service as type load balancer?
Can anyone provide an example of this, thanks