Closed sudheersagi closed 2 years ago
Hi @sudheersagi,
We do manage the configuration through environment variables as described in the mongodb-sharded container repository.
When you override the configuration files mountpoint, you break some of our logic and you should make sure you understand how that impacts the container.
You can find more information in this repository:
If you use only our values, you should be able to get the same result without overriding the configuration mount point.
Could you confirm you are able to connect to configsvr1.example.com
? How can the pod resolve this host name?
I'm happy to help further, but I think there are a couple of things that should be reviewed to make sure the problem is indeed coming from the chart.
Thank you so much @yilmi for helping me here.
Could you confirm you are able to connect to configsvr1.example.com? How can the pod resolve this host name?
I tried to ran below command from the pod where mongos is setting up
mongos --configdb cfgrs0/configsvr0.example.com:27019 --port 27017 --clusterAuthMode keyFile --keyFile /opt/bitnami/mongodb/conf/keyfile
Here are the logs
{"t":{"$date":"2021-11-01T11:34:34.507Z"},"s":"W", "c":"SHARDING", "id":24132, "ctx":"main","msg":"Running a sharded cluster with fewer than 3 config servers should only be done for testing purposes and is not recommended for production."}
{"t":{"$date":"2021-11-01T11:34:34.508+00:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"main","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
{"t":{"$date":"2021-11-01T11:34:34.509+00:00"},"s":"W", "c":"CONTROL", "id":22138, "ctx":"main","msg":"You are running this process as the root user, which is not recommended","tags":["startupWarnings"]}
{"t":{"$date":"2021-11-01T11:34:34.509+00:00"},"s":"W", "c":"CONTROL", "id":22140, "ctx":"main","msg":"This server is bound to localhost. Remote systems will be unable to connect to this server. Start the server with --bind_ip <address> to specify which IP addresses it should serve responses from, or with --bind_ip_all to bind to all interfaces. If this behavior is desired, start the server with --bind_ip 127.0.0.1 to disable this warning","tags":["startupWarnings"]}
{"t":{"$date":"2021-11-01T11:34:34.574+00:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"mongosMain","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.10","gitVersion":"58971da1ef93435a9f62bf4708a81713def6e88c","openSSLVersion":"OpenSSL 1.1.1d 10 Sep 2019","modules":[],"allocator":"tcmalloc","environment":{"distmod":"debian10","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2021-11-01T11:34:34.574+00:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"mongosMain","msg":"Operating System","attr":{"os":{"name":"PRETTY_NAME=\"Debian GNU/Linux 10 (buster)\"","version":"Kernel 5.4.149-73.259.amzn2.x86_64"}}}
{"t":{"$date":"2021-11-01T11:34:34.574+00:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"mongosMain","msg":"Options set by command line","attr":{"options":{"net":{"port":27017},"security":{"clusterAuthMode":"keyFile","keyFile":"/opt/bitnami/mongodb/conf/keyfile"},"sharding":{"configDB":"cfgrs0/configsvr0.example.com:27019"}}}}
{"t":{"$date":"2021-11-01T11:34:34.574+00:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"mongosMain","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
{"t":{"$date":"2021-11-01T11:34:34.575+00:00"},"s":"I", "c":"NETWORK", "id":4603701, "ctx":"mongosMain","msg":"Starting Replica Set Monitor","attr":{"protocol":"streamable","uri":"cfgrs0/configsvr0.example.com:27019"}}
{"t":{"$date":"2021-11-01T11:34:34.575+00:00"},"s":"I", "c":"-", "id":4333223, "ctx":"mongosMain","msg":"RSM now monitoring replica set","attr":{"replicaSet":"cfgrs0","nReplicaSetMembers":1}}
{"t":{"$date":"2021-11-01T11:34:34.575+00:00"},"s":"I", "c":"-", "id":4333226, "ctx":"mongosMain","msg":"RSM host was added to the topology","attr":{"replicaSet":"cfgrs0","host":"configsvr0.example.com:27019"}}
{"t":{"$date":"2021-11-01T11:34:34.575+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Connecting","attr":{"hostAndPort":"configsvr0.example.com:27019"}}
{"t":{"$date":"2021-11-01T11:34:34.575+00:00"},"s":"I", "c":"SHARDING", "id":22649, "ctx":"thread1","msg":"Creating distributed lock ping thread","attr":{"processId":"mongos-mongos-848f6d7cd7-ftc7f:27017:1635766474:6736574883495560854","pingIntervalMillis":30000}}
{"t":{"$date":"2021-11-01T11:34:34.581+00:00"},"s":"I", "c":"NETWORK", "id":23729, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"ServerPingMonitor is now monitoring host","attr":{"host":"configsvr0.example.com:27019","replicaSet":"cfgrs0"}}
{"t":{"$date":"2021-11-01T11:34:34.581+00:00"},"s":"I", "c":"NETWORK", "id":4333213, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM Topology Change","attr":{"replicaSet":"cfgrs0","newTopologyDescription":"{ id: \"569a2ee3-f369-4c97-8df9-fefa1e26a98e\", topologyType: \"ReplicaSetNoPrimary\", servers: { configsvr0.example.com:27019: { address: \"configsvr0.example.com:27019\", topologyVersion: { processId: ObjectId('6166610c90e7cec7b72108a5'), counter: 4 }, roundTripTime: 633, lastWriteDate: new Date(1635766473000), opTime: { ts: Timestamp(1635766473, 3), t: 1 }, type: \"RSSecondary\", minWireVersion: 9, maxWireVersion: 9, me: \"configsvr0.example.com:27019\", setName: \"cfgrs0\", setVersion: 4, primary: \"configsvr1.example.com:27019\", lastUpdateTime: new Date(1635766474581), logicalSessionTimeoutMinutes: 30, hosts: { 0: \"configsvr1.example.com:27019\", 1: \"configsvr2.example.com:27019\", 2: \"configsvr0.example.com:27019\" }, arbiters: {}, passives: {} }, configsvr1.example.com:27019: { address: \"configsvr1.example.com:27019\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} }, configsvr2.example.com:27019: { address: \"configsvr2.example.com:27019\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, logicalSessionTimeoutMinutes: 30, setName: \"cfgrs0\", compatible: true }","previousTopologyDescription":"{ id: \"b5e13d07-fc75-4dbc-90e3-2816e624de3f\", topologyType: \"ReplicaSetNoPrimary\", servers: { configsvr0.example.com:27019: { address: \"configsvr0.example.com:27019\", type: \"Unknown\", minWireVersion: 0, maxWireVersion: 0, lastUpdateTime: new Date(-9223372036854775808), hosts: {}, arbiters: {}, passives: {} } }, setName: \"cfgrs0\", compatible: true }"}}
{"t":{"$date":"2021-11-01T11:34:34.581+00:00"},"s":"I", "c":"-", "id":4333226, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host was added to the topology","attr":{"replicaSet":"cfgrs0","host":"configsvr1.example.com:27019"}}
{"t":{"$date":"2021-11-01T11:34:34.581+00:00"},"s":"I", "c":"-", "id":4333226, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"RSM host was added to the topology","attr":{"replicaSet":"cfgrs0","host":"configsvr2.example.com:27019"}}
{"t":{"$date":"2021-11-01T11:34:34.581+00:00"},"s":"I", "c":"-", "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"cfgrs0","host":"configsvr2.example.com:27019","delayMillis":0}}
{"t":{"$date":"2021-11-01T11:34:34.581+00:00"},"s":"I", "c":"-", "id":4333218, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Rescheduling the next replica set monitoring request","attr":{"replicaSet":"cfgrs0","host":"configsvr1.example.com:27019","delayMillis":0}}
{"t":{"$date":"2021-11-01T11:34:34.581+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Connecting","attr":{"hostAndPort":"configsvr2.example.com:27019"}}
{"t":{"$date":"2021-11-01T11:34:34.581+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Connecting","attr":{"hostAndPort":"configsvr1.example.com:27019"}}
{"t":{"$date":"2021-11-01T11:34:34.649+00:00"},"s":"I", "c":"SHARDING", "id":22792, "ctx":"ShardRegistry","msg":"Term advanced for config server","attr":{"opTime":{"ts":{"$timestamp":{"t":1635766473,"i":3}},"t":1},"prevOpTime":{"ts":{"$timestamp":{"t":0,"i":0}},"t":-1},"reason":"reply from config server node","clientAddress":"(unknown)"}}
{"t":{"$date":"2021-11-01T11:34:34.649+00:00"},"s":"I", "c":"NETWORK", "id":4603701, "ctx":"shard-registry-reload","msg":"Starting Replica Set Monitor","attr":{"protocol":"streamable","uri":"shardrs1/shardsvr1-1.example.com:27018,shardsvr1-2.example.com:27018,shardsvr1-3.example.com:27018"}}
Regarding overrides, initially i tried to use your values without any overrides but the config are not copied, and i see errors like confsvr host (127.0.0.1) unknown so after MONGODB_MOUNTED_CONFDIR overriding i see custom config gets copied to /opt/bitnami/mongodb/conf, from mounted path /bitnami/mongodb/conf_
Sorry, if my observation is wrong. Also I'm going with same deployment without any changes in volume mount.
i think by default MONGODB_MOUNTED_CONFDIR setting with /bitnami/conf
where in deployment yaml the conf is mounted to /bitnami/mongodb/conf_
- name: config
mountPath: /bitnami/mongodb/conf/
Thanks for the additional information here. Ok, regarding this issue you mentioned below:
Regarding overrides, initially i tried to use your values without any overrides but the config are not copied, and i see errors like confsvr host (127.0.0.1) unknown so after MONGODB_MOUNTED_CONF_DIR overriding i see custom config gets copied to /opt/bitnami/mongodb/conf, from mounted path /bitnami/mongodb/conf
Please check if there are any pvc's left after uninstalling your chart as helm won't delete them for you. This would be an issue, as when the container starts, it will check if some files exist already, and if so will skip some configuration steps. So that's probably what happened (happens quite often in our issues).
Regarding the connectivity issue you report, have you noticed the following log line?
{"t":{"$date":"2021-11-01T11:34:34.581+00:00"},"s":"I", "c":"CONNPOOL", "id":22576, "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Connecting","attr":{"hostAndPort":"configsvr1.example.com:27019"}} {"t":{"$date":"2021-11-01T11:34:34.649+00:00"},"s":"I", "c":"SHARDING", "id":22792, "ctx":"ShardRegistry","msg":"Term advanced for config server","attr":{"opTime":{"ts":{"$timestamp":{"t":1635766473,"i":3}},"t":1},"prevOpTime":{"ts":{"$timestamp":{"t":0,"i":0}},"t":-1},"reason":"reply from config server node","clientAddress":"(unknown)"}}
The second line mentions "clientAddress" as "unkown" which could be the cause of your issues here. Could you just try performing a curl -v https://configsvr0.example.com:27019
or similar (perhaps with http)?
This should at least give you some ideas about whether or not the TCP handshake is going through, and if you're using TLS, if the Authentication and Key Exchange parts are also successful.
Thanks for pointing this @yilmi
Here is the curl output
curl -v https://configsvr0.example.com:27019
Expire in 0 ms for 6 (transfer 0x55cf28e45fb0)
* Expire in 1 ms for 1 (transfer 0x55cf28e45fb0)
* Expire in 0 ms for 1 (transfer 0x55cf28e45fb0)
* Expire in 1 ms for 1 (transfer 0x55cf28e45fb0)
* Expire in 0 ms for 1 (transfer 0x55cf28e45fb0)
* Expire in 0 ms for 1 (transfer 0x55cf28e45fb0)
* Expire in 1 ms for 1 (transfer 0x55cf28e45fb0)
* Expire in 0 ms for 1 (transfer 0x55cf28e45fb0)
* Expire in 0 ms for 1 (transfer 0x55cf28e45fb0)
* Expire in 1 ms for 1 (transfer 0x55cf28e45fb0)
...
...
...
* Expire in 9 ms for 1 (transfer 0x55cf28e45fb0)
* Expire in 9 ms for 1 (transfer 0x55cf28e45fb0)
* Expire in 12 ms for 1 (transfer 0x55cf28e45fb0)
* Trying 10.xxx.8x.2x...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x55cf28e45fb0)
* Connected to configsvr0.example.com (10.xxx.8x.2x) port 27019 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to configsvr0.example.com:27019
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to configsvr0.example.com:27019
Regarding overrides, i read about this pvc's thing in one of issue comments and i just make sure to not have any PVC's left in namespace,Once again i will cross verify it. We have pvc's in different k8s namespace(test env) within the same cluster and i hope that doesn't causes any issue.
Hi @yilmi , we are not using any SSL on config server both are running within same network, just connecting using keyFile /opt/bitnami/mongodb/conf/keyfile Also is there anything in values.yaml to control the non-secure connection from mongos to configsvr.
"clientAddress" as "unkown"
It would be great if you can add more details around it.
Hi @sudheersagi, I'm not trying to give you precise troubleshooting steps, I'm just trying to give you some pointers.
We are not MongoDB developers, and right now your problem seems more related to how MongoDB works than our chart itself. It seems that the Config Server also needs connectivity with the mongos - https://dba.stackexchange.com/a/201698
On our end, the values provide support for defining the config server, please reopen this issue if you have an issue with the chart.
Hi @yilmi
I believe this is for sure not a problem from the MONGODB side but from the bitnami chart configuration itself because, when we tried to overwrite the /opt/bitnami/mongodb/conf/mongos.conf with the correct port configuration & fed it to the mongodb startup scripts, the MONGODB connection succeeded.
We are looking forward for a couple of mins of your time (may be a quick zoom call) to talk about our suspicion on the 'PORT variables overwrite not happening'.
#######################################
More details are described below:
I'm able to connect to Mongo configsvr from the POD without any issue if we run manually. What i was trying to tell you is after installed the chart, as soon the container is up and running it should create mongos process and and listen on port(here it is 27017) for any incoming requests, which is not happening because I dont see any pid inside /opt/bitnami/mongodb/tmp/. Correct me if this is not the expected behaviour of chart.
Just to make sure there are no existing PVCs
kubectl get pvc --namespace namespace1
No resources found in namespace1 namespace.
I followed the below steps to make sure there is not connectivity issue with MongoDB Config Server.
Step1: Installed the chart with above values.yaml file. Step2: Exec to POD and verified conf files with custom changes
- /bitnami/mongodb/conf/mongodb.conf -- contains custom config as given in 'config' variable from yaml - /opt/bitnami/mongodb/conf/keyfile -- Copied from 'replicasetKey' variable - /opt/bitnami/mongodb/conf/mongodb.conf -- same as in 'config' variable - /opt/bitnami/mongodb/conf/mongos.conf -- same as in 'config' variable - /opt/bitnami/mongodb/tmp/ -- process id is missing, assuming mongos not started
Step3: Perform command mongos -f
mongos -f /opt/bitnami/mongodb/conf/mongos.conf about to fork child process, waiting until server is ready for connections. forked process: 747 child process started successfully, parent exiting
Step4: verify the process id, now i see pid is created which is supposed to be created during container startup itself.
/opt/bitnami/mongodb/tmp# ls mongodb-27017.sock mongodb.pid
Step5: Connect (external)configsvr/shrdsvr from mongos running pod.In step4 we see mongos process is started
$mongo --host 127.0.0.1 --port 27017 --authenticationDatabase "admin" -u "user" -p "passwd" MongoDB shell version v4.4.10 connecting to: mongodb://127.0.0.1:27017/?authSource=admin&compressors=disabled&gssapiServiceName=mongodb Implicit session: session { "id" : UUID("066431fe-028a-453e-b689-f703a5fc781c") } MongoDB server version: 4.4.10 mongos>
FYI, Step2 to Step5 were performed within mongos pod. Step6: Tried to connect from application and verify the data from MongoDB.
logger:org.mongodb.driver.connection message:Opened connection [connectionId{localValue:13, serverValue:85}] to mongodb.namespace1.svc.cluster.local:27017
From the above steps, i believe there is no connectivity issue with Mongo configsvr that is running externally and i think the mongos process should be created when the contianer up and running (if this is the expected behaiour of this chart).
I did grep on process that are running when POD is available
# ps -ef
root 1 0 0 17:57 ? 00:00:00 /bin/bash /opt/bitnami/scripts/mongodb-sharded/entrypoint.sh /opt/bitnami/scripts/mongodb-sharded/run.sh
root 13 1 0 17:57 ? 00:00:00 /bin/bash /opt/bitnami/scripts/mongodb-sharded/setup.sh
I also tried by launching separate instance(within network outside cluster) and manually installed mongodb binary with same mongos conf file and able to connect our centralised mongo configsvr without any issues. We still see
"clientAddress":"(unknown)"
in the log but able to connect mongo. So i guess this is not actual error and it is ignorable
Which chart: mongodb-sharded appVersion 4.4.10 version 3.9.14
Describe the bug I am running the chart on AWS EKS. Our architecture is in such a way that we want to run only mongos within our Application cluster as a service and connecting to external config server.Here our config server and shard data are centralised and running in another cluster within network which is common for multiple services.
I deployed mongodb-sharded chart and provide the necessary details in values.yaml as mentioned in README section
https://github.com/bitnami/charts/tree/master/bitnami/mongodb-sharded#using-an-external-config-server
. Also Attached the values yaml for your reference and logs.Mongos Logs
Values.yaml
Reason for adding below variable is to copy the
config
to /opt/bitnami/mongodb/conf/mongos.conf, mongodb.conf otherwise it is creating conf with default values and it is taking configsvr port as 27017 as default where in our case the config server run at port 27019.When i looked into the container pod, the provided 'config' details are copied to mongodb.conf and mongos.conf from mount path /bitnami/mongodb/conf, From the pod logs i believe it is establishing connection to config server but when my app connect to mongos service i get 'connection refused' error from application end.
I tried by enabling diagnosticMode and verified logs still no use.Not clear what am i missing. Please help me in fixing it.
To Reproduce helm -upgrade install mongos --namespace namespace1 mongodb-sharded
Expected behavior Mongos pid should be created /opt/bitnami/mongodb/tmp/mongodb.pid and application is able to talk to mongos.
Version of Helm and Kubernetes:
helm version
:kubectl version
:Additional context Tried to run container as root user also to check the listening ports and observed port 27017 is not available for access from application pod.