some pods stuck in init state indefinitely

[haydenp-zhou]

Hello! I was trying to deploy openwhisk in docker-desktop following documentation steps,But owdev-alarmprovider,owdev-controller and nginx pods are stuck in PodInitializing phase. Please find details as below:

>>kubectl logs owdev-install-packages-v5hm8 -c install-packages -n openwhisk
Error from server (BadRequest): container "install-packages" in pod "owdev-install-packages-v5hm8" is waiting to start: PodInitializing
>>kubectl get all -n openwhisk
NAME                                       READY   STATUS      RESTARTS   AGE
pod/owdev-alarmprovider-5b86cb64ff-lkgc7   0/1     Init:0/1    0          19m
pod/owdev-apigateway-bccbbcd67-fvwsz       1/1     Running     0          19m
pod/owdev-controller-0                     0/1     Init:1/2    0          19m
pod/owdev-couchdb-584676b956-5n8tf         1/1     Running     0          19m
pod/owdev-gen-certs-626bp                  0/1     Completed   0          19m
pod/owdev-init-couchdb-5mpzl               1/1     Running     0          19m
pod/owdev-install-packages-v5hm8           0/1     Init:0/1    0          19m
pod/owdev-invoker-0                        0/1     Init:0/1    0          19m
pod/owdev-kafka-0                          1/1     Running     0          19m
pod/owdev-kafkaprovider-5574d4bf5f-6j9q7   0/1     Init:0/1    0          19m
pod/owdev-nginx-86749d59cb-sh9vr           0/1     Init:0/1    0          19m
pod/owdev-redis-d65649c5b-z4xk8            1/1     Running     0          19m
pod/owdev-wskadmin                         1/1     Running     0          19m
pod/owdev-zookeeper-0                      1/1     Running     0          19m

NAME                       TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
service/owdev-apigateway   ClusterIP   10.101.42.56    <none>        8080/TCP,9000/TCP            19m
service/owdev-controller   ClusterIP   10.107.156.5    <none>        8080/TCP                     19m
service/owdev-couchdb      ClusterIP   10.99.0.169     <none>        5984/TCP                     19m
service/owdev-kafka        ClusterIP   None            <none>        9092/TCP                     19m
service/owdev-nginx        NodePort    10.111.89.225   <none>        80:30853/TCP,443:31001/TCP   19m
service/owdev-redis        ClusterIP   10.109.26.150   <none>        6379/TCP                     19m
service/owdev-zookeeper    ClusterIP   None            <none>        2181/TCP,2888/TCP,3888/TCP   19m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/owdev-alarmprovider   0/1     1            0           19m
deployment.apps/owdev-apigateway      1/1     1            1           19m
deployment.apps/owdev-couchdb         1/1     1            1           19m
deployment.apps/owdev-kafkaprovider   0/1     1            0           19m
deployment.apps/owdev-nginx           0/1     1            0           19m
deployment.apps/owdev-redis           1/1     1            1           19m

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/owdev-alarmprovider-5b86cb64ff   1         1         0       19m
replicaset.apps/owdev-apigateway-bccbbcd67       1         1         1       19m
replicaset.apps/owdev-couchdb-584676b956         1         1         1       19m
replicaset.apps/owdev-kafkaprovider-5574d4bf5f   1         1         0       19m
replicaset.apps/owdev-nginx-86749d59cb           1         1         0       19m
replicaset.apps/owdev-redis-d65649c5b            1         1         1       19m

NAME                                READY   AGE
statefulset.apps/owdev-controller   0/1     19m
statefulset.apps/owdev-invoker      0/1     19m
statefulset.apps/owdev-kafka        1/1     19m
statefulset.apps/owdev-zookeeper    1/1     19m

NAME                               COMPLETIONS   DURATION   AGE
job.batch/owdev-gen-certs          1/1           12s        19m
job.batch/owdev-init-couchdb       0/1           19m        19m
job.batch/owdev-install-packages   0/1           19m        19m

Please suggest if I am doing something wrong.

[dgrove-oss]

I'd probably investigate why the nginx pod isn't reaching a ready state first. There's not much going on with nginx that is openwhisk specific, so it probably points to a general problem on your cluster with pulling images, or not having enough resources, or something similar.

[dgrove-oss]

Strike the above. I forgot nginx had an init pod that is configured to wait for the controller to be up. So the place to look is that invoker pod.

[rabbah]

I suspect - this has to do with networking. The pods stuck in init are ones that need to reach other pods:

kafka depends on zk controller | invoker depend on kafka and couch

is it possible the networking isn't properly set up?

[haydenp-zhou]

couchdb can't clone openwhisk

kubectl logs owdev-init-couchdb-5xx4b -n openwhisk Cloning into '/openwhisk'... error: RPC failed; curl 18 transfer closed with outstanding read data remaining fatal: The remote end hung up unexpectedly fatal: early EOF fatal: index-pack failed kubectl get pods -n openwhisk NAME READY STATUS RESTARTS AGE owdev-alarmprovider-5b86cb64ff-g2kqc 0/1 Init:0/1 0 93m owdev-apigateway-bccbbcd67-n8c8f 1/1 Running 0 93m owdev-controller-0 0/1 Init:1/2 0 93m owdev-couchdb-584676b956-rmgh5 1/1 Running 0 93m owdev-gen-certs-6mb4s 0/1 Completed 0 93m owdev-init-couchdb-5xx4b 0/1 Error 0 62m owdev-init-couchdb-r7bgp 0/1 Error 0 67m owdev-init-couchdb-r8k4p 0/1 Error 0 93m owdev-init-couchdb-zqb6j 0/1 Error 0 81m owdev-install-packages-chphp 0/1 Init:0/1 0 93m owdev-invoker-0 0/1 Init:0/1 0 93m owdev-kafka-0 1/1 Running 0 93m owdev-kafkaprovider-5574d4bf5f-45wwj 0/1 Init:0/1 0 93m owdev-nginx-86749d59cb-thrfx 0/1 Init:0/1 0 93m owdev-redis-d65649c5b-kxwds 1/1 Running 0 93m owdev-wskadmin 1/1 Running 0 93m owdev-zookeeper-0 1/1 Running 0 93m

[pearPLUS]

I have the same issue in the initialization phase of Couchdb pod.couchdb can't clone openwhisk. have u solved this problem?

kubectl logs owdev-init-couchdb-zq5qs -n openwhisk Cloning into '/openwhisk'... fatal: unable to access 'https://github.com/apache/openwhisk/': Could not resolve host: github.com