KrishnaKoppineni
commented
5 years ago - Due to the Weavenet issue the pods are suddenly not working and gone to CreashLoopBackOff state
[root@prod-app1 ~]# kubectl get pod -n capiot
NAME READY STATUS RESTARTS AGE
b2b-6d44c5d847-82zgp 1/1 Running 0 28h
b2bgw-66f9f95c85-j9pxf 0/1 CrashLoopBackOff 12 4h12m
dm-5cbd787874-bp6ld 1/1 Running 0 28h
gw-59f69487f7-8f7j8 0/1 CrashLoopBackOff 7 28h
mon-5fb8d485cb-jrt76 0/1 CrashLoopBackOff 8 28h
nats-c666bb65b-qlg6g 1/1 Running 0 28h
ne-f9b6468cc-tx74j 0/1 CrashLoopBackOff 7 28h
nginx-859c9759f8-2jqj6 1/1 Running 0 28h
pm-64f765f7f5-fngbw 0/1 CrashLoopBackOff 7 28h
redis-547fdbb749-nfw5r 1/1 Running 0 28h
sec-5bcb5f85fb-752xv 0/1 CrashLoopBackOff 7 4h12m
sm-59677d5bcc-l72g8 0/1 Running 7 28h
user-6c886f554-vcwl2 0/1 CrashLoopBackOff 8 28h
wf-d85b7c498-gpkpp 0/1 CrashLoopBackOff 9 28h
[root@prod-app1 ~]#When look into one of the pod logs it gives the following error "Not able to connect to Mongodb"
> **[root@prod-app1 ~]# kubectl logs -f -n capiot sec-5bcb5f85fb-752xv
WARNING: No configurations found in configuration directory:/app/config
WARNING: To disable this warning set SUPPRESS_NO_CONFIG_WARNING in the environment.
[2019-09-04T12:12:57.794] [INFO] [security] [sec-5bcb5f85fb-752xv] - Server started on port 10007
[2019-09-04T12:13:02.443] [ERROR] [odp-utils-nats-streaming] - Could not connect to server: Error: getaddrinfo EAI_AGAIN nats.capiot:4222
[2019-09-04T12:13:02.577] [ERROR] security [sec-5bcb5f85fb-752xv] - ERROR :: Unable to connect to Kubernetes API server
[2019-09-04T12:13:02.579] [INFO] security [sec-5bcb5f85fb-752xv] -
[2019-09-04T12:13:13.642] [ERROR] [security] [sec-5bcb5f85fb-752xv] - ------------------------- Database connection lost -------------------------
[2019-09-04T12:13:13.644] [ERROR] [security] [sec-5bcb5f85fb-752xv] - { MongoNetworkError: failed to connect to server [172.31.1.231:27019] on first connect [MongoNetworkError: connect EHOSTUNREACH 172.31.1.231:27019]
at Pool.<anonymous> (/app/node_modules/mongodb-core/lib/topologies/server.js:564:11)
at emitOne (events.js:116:13)
at Pool.emit (events.js:211:7)
at Connection.<anonymous> (/app/node_modules/mongodb-core/lib/connection/pool.js:317:12)
at Object.onceWrapper (events.js:317:30)
at emitTwo (events.js:126:13)
at Connection.emit (events.js:214:7)
at Socket.<anonymous> (/app/node_modules/mongodb-core/lib/connection/connection.js:246:50)
at Object.onceWrapper (events.js:315:30)
at emitOne (events.js:116:13)
at Socket.emit (events.js:211:7)
at emitErrorNT (internal/streams/destroy.js:66:8)
at _combinedTickCallback (internal/process/next_tick.js:139:11)
at process._tickDomainCallback (internal/process/next_tick.js:219:9)
name: 'MongoNetworkError',
errorLabels: [ 'TransientTransactionError' ],
[Symbol(mongoErrorContextSymbol)]: {} }
[root@prod-app1 ~]#**But I am able to telnet from the host machine.
> **[root@prod-app1 ~]# curl http://172.31.1.231:27017
It looks like you are trying to access MongoDB over HTTP on the native driver port.
[root@prod-app1 ~]#**After Restarting the weavenet pod it is working fine and the application pods are able to connect the mongodb......But this issue is getting resolved for a specific amount of time only. After some using the again it is the same issue.
[root@prod-app1 ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-5c98db65d4-7xqff 1/1 Running 4 5d5h
coredns-5c98db65d4-wpkkl 1/1 Running 4 5d5h
etcd-prod-app1.gati.com 1/1 Running 4 5d5h
kube-apiserver-prod-app1.gati.com 1/1 Running 4 5d5h
kube-controller-manager-prod-app1.gati.com 1/1 Running 4 5d5h
kube-proxy-g8p52 1/1 Running 1 5d4h
kube-proxy-wt2kr 1/1 Running 4 5d5h
kube-scheduler-prod-app1.gati.com 1/1 Running 4 5d5h
weave-net-qcftw 2/2 Running 0 4h14m[root@prod-app1 ~]# kubectl delete pod -n kube-system weave-net-qcftw
pod "weave-net-qcftw" deleted
[root@prod-app1 ~]#
Now all the pods are in Running state issue resolved for this movement. It will come again after sometime..
[root@prod-app1 ~]# kubectl get pod -n capiot
NAME READY STATUS RESTARTS AGE
b2b-6d44c5d847-j7fnr 1/1 Running 0 95s
b2bgw-66f9f95c85-qlzn8 1/1 Running 3 95s
dm-5cbd787874-c6n8f 1/1 Running 0 95s
gw-59f69487f7-qlskd 1/1 Running 0 95s
mon-5fb8d485cb-hmsms 1/1 Running 0 95s
nats-c666bb65b-gwc7z 1/1 Running 0 95s
ne-f9b6468cc-z9mfn 1/1 Running 0 95s
nginx-859c9759f8-28jlq 1/1 Running 0 95s
pm-64f765f7f5-ghz8b 1/1 Running 0 95s
redis-547fdbb749-pfhn9 1/1 Running 0 94s
sec-5bcb5f85fb-4w4nn 1/1 Running 0 94s
sm-59677d5bcc-7jgwz 1/1 Running 0 94s
user-6c886f554-z629v 1/1 Running 0 94s
wf-d85b7c498-nx2lv 1/1 Running 1 94sSo, please can anyone help me to resolve the issue. Thank you.
murali-reddy
bboreham
HI, I am running simple Kubernetes cluster with single node acts as both (Master & worker) as well....... 1). In that I am facing an issue with Weanet Networking in kubernetes. Suddenly my pods are not able to communicate mongoDB running on another host and both are in same zone......But i am able telnet from the server.
2) After restarting the weavenet pod only the application pods are able to connect mongodb.
K8s version: 1.15 Weavnet Version: 2.5.2 uname -a: Linux prod-app1.gati.com 3.10.0-862.el7.x86_64 Docker version: 18.09.7
Can anyone please help me to resolve the issue. Thank you