"Failed to connect to 127.0.0.1:27017" after the pods are transferred to a new node in GKE

[hiowenluke]

Hi Paul,

The gke-mongodb-shards-demo is great, and I had applied it in my GKE, thank you very much.

Now I added a new node in kubernetes, and transferred all pods (including the mongodb shards pods) to it via labeled the old node as taint. After that, I found that my app can not connect to this mongodb shards.

kubectl logs mongos-router-0 -n mongodb

2018-05-15T08:53:51.757+0000 I NETWORK  [shard registry reload] Successfully connected to mongod-configdb-1.mongodb-configdb-service.mongodb.svc.cluster.local:27017 (1 connections now open to mongod-configdb-1.mongodb-configdb-service.mongodb.svc.cluster.local:27017 with a 5 second timeout)
2018-05-15T08:53:51.851+0000 I NETWORK  [initandlisten] Successfully connected to mongod-configdb-0.mongodb-configdb-service.mongodb.svc.cluster.local:27017 (1 connections now open to mongod-configdb-0.mongodb-configdb-service.mongodb.svc.cluster.local:27017 with a 5 second timeout)

2018-05-15T08:53:51.851+0000 W NETWORK  [initandlisten] Unable to reach primary for set ConfigDBRepSet
2018-05-15T08:53:51.851+0000 I NETWORK  [initandlisten] Cannot reach any nodes for set ConfigDBRepSet. Please check network connectivity and the status of the set. This has happened for 1 checks in a row.

When I re-generate the mongodb shards, the following error occured repeatedly :

Waiting for all the shards and configdb containers to come up (Tue May 15 17:16:58 CST 2018)...
 (IGNORE any reported not found & connection errors)
  connection to 127.0.0.1:27017
  2018-05-15T09:17:35.750+0000 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused
2018-05-15T09:17:35.750+0000 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:251:13
@(connect):1:21
exception: connect failed
command terminated with exit code 1

What should I do ? Thanks a lot.

Best, Owen

[jimmiebtlr]

Believe what your seeing is probably this

https://github.com/pkdone/gke-mongodb-shards-demo/issues/7

[hiowenluke]

Thank you so much

[Mantsje]

I saw that in the current version these values from the other issue have changed: Config server writes to /data/configdb I believe rather than /data/db. However I'm still encountering the connection refused errors after restarting the system.