apache / openwhisk-deploy-kube

The Apache OpenWhisk Kubernetes Deployment repository supports deploying the Apache OpenWhisk system on Kubernetes and OpenShift clusters.
https://openwhisk.apache.org/
Apache License 2.0
302 stars 232 forks source link

controller and invoker pods are stuck in PodInitializing #311

Closed saurabh1284 closed 6 years ago

saurabh1284 commented 6 years ago

I have an kubernetes cluster of 2 nodes (1 master and 1 worker) and was trying to deploy openwhisk following documentation steps. But controller and invoker pods are stuck in PodInitializing phase. I am using latest clone. Please find details as below:

-> kubectl get nodes --show-labels NAME STATUS ROLES AGE VERSION LABELS server-f2-02.ipc3dev.com Ready master 33d v1.11.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=server-f2-02.ipc3dev.com,node-role.kubernetes.io/master=master server-f2-0f.ipc3dev.com Ready worker 33d v1.11.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=server-f2-0f.ipc3dev.com,node-role.kubernetes.io/worker=worker,openwhisk-role=invoker

-> kubectl describe nodes | grep InternalIP nternalIP: 10.71.3.7 InternalIP: 10.71.3.8

-> cat mycluster.yaml whisk: ingress: type: NodePort api_host_name: 10.71.3.7 api_host_port: 31001 nginx: httpsNodePort: 31001

-> kubectl -n openwhisk get pods NAME READY STATUS RESTARTS AGE apigateway-64b5fb47df-h8v5f 1/1 Running 0 2h controller-0 0/1 Init:1/2 0 2h couchdb-6db7d989dd-886mc 1/1 Running 0 2h invoker-b5mpt 0/1 Init:1/2 0 2h kafka-0 1/1 Running 0 2h nginx-55d45778cb-5vgd6 1/1 Running 0 2h redis-6788bb4875-lcwgw 1/1 Running 0 2h zookeeper-0 1/1 Running 0 2h

-> kubectl -n openwhisk describe pod controller-0 Name: controller-0 Namespace: openwhisk Node: server-f2-0f.ipc3dev.com/10.71.3.8 Start Time: Mon, 08 Oct 2018 04:42:25 -0400 Labels: controller-revision-hash=controller-5bc6758f68 name=controller statefulset.kubernetes.io/pod-name=controller-0 Annotations: Status: Pending IP: 10.244.1.156 Controlled By: StatefulSet/controller Init Containers: wait-for-kafka: Container ID: docker://6d870e1e10ab5913fd8f48c00f7ee342c7e8e9e0af25a727a15d2229ccb4d00a Image: busybox Image ID: docker-pullable://docker.io/busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd Port: Host Port: Command: sh -c result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 zookeeper-0.zookeeper.openwhisk.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi done; echo "Zookeeper is up; will wait for 10 seconds to give kafka time to initialize"; sleep 10; State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 08 Oct 2018 04:42:37 -0400 Finished: Mon, 08 Oct 2018 04:43:17 -0400 Ready: True Restart Count: 0 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9qfsx (ro) wait-for-couchdb: Container ID: docker://08391dc6b3eb76127050c35f0948006a0e398d7b761dc208f83c2b621f4a5c64 Image: busybox Image ID: docker-pullable://docker.io/busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd Port: Host Port: Command: sh -c while true; do echo 'checking CouchDB readiness'; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; if [ $result -eq 0 ]; then echo 'Success: CouchDB is ready!'; break; fi; echo '...not ready yet; sleeping 3 seconds before retry'; sleep 3; done; State: Running Started: Mon, 08 Oct 2018 04:43:19 -0400 Ready: False Restart Count: 0 Environment: READINESS_URL: http://couchdb.openwhisk.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker Mounts: /var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9qfsx (ro) Containers: controller: Container ID: Image: openwhisk/controller:latest Image ID: Ports: 8080/TCP, 2552/TCP, 19999/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Command: /bin/bash -c /init.sh hostname | cut -d'-' -f2 State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Liveness: http-get http://:8080/ping delay=5s timeout=1s period=10s #success=1 #failure=3 Environment: PORT: 8080 CONFIG_whisk_info_date: <set to the key 'whisk_info_date' of config map 'whisk.config'> Optional: false CONFIG_whisk_info_buildNo: <set to the key 'whisk_info_buildNo' of config map 'whisk.config'> Optional: false JAVA_OPTS: -Xmx1024M CONTROLLER_OPTS: RUNTIMES_MANIFEST: { "runtimes": { "nodejs": [ { "kind": "nodejs", "image": { "prefix": "openwhisk", "name": "nodejsaction", "tag": "latest" }, "deprecated": true }, { "kind": "nodejs:6", "default": true, "image": { "prefix": "openwhisk", "name": "nodejs6action", "tag": "latest" }, "deprecated": false, "stemCells": [{ "count": 2, "memory": "256 MB" }] }, { "kind": "nodejs:8", "default": false, "image": { "prefix": "openwhisk", "name": "action-nodejs-v8", "tag": "latest" }, "deprecated": false } ], "python": [ { "kind": "python", "image": { "prefix": "openwhisk", "name": "python2action", "tag": "latest" }, "deprecated": false }, { "kind": "python:2", "default": true, "image": { "prefix": "openwhisk", "name": "python2action", "tag": "latest" }, "deprecated": false }, { "kind": "python:3", "image": { "prefix": "openwhisk", "name": "python3action", "tag": "latest" }, "deprecated": false } ], "swift": [ { "kind": "swift:3.1.1", "image": { "prefix": "openwhisk", "name": "action-swift-v3.1.1", "tag": "latest" }, "deprecated": false }, { "kind": "swift:4.1", "default": true, "image": { "prefix": "openwhisk", "name": "action-swift-v4.1", "tag": "latest" }, "deprecated": false } ], "java": [ { "kind": "java", "default": true, "image": { "prefix": "openwhisk", "name": "java8action", "tag": "latest" }, "deprecated": false, "attached": { "attachmentName": "jarfile", "attachmentType": "application/java-archive" }, "sentinelledLogs": false, "requireMain": true } ], "php": [ { "kind": "php:7.1", "default": true, "deprecated": false, "image": { "prefix": "openwhisk", "name": "action-php-v7.1", "tag": "latest" } } ], "ruby": [ { "kind": "ruby:2.5", "default": true, "deprecated": false, "image": { "prefix": "openwhisk", "name": "action-ruby-v2.5", "tag": "latest" } } ] }, "blackboxes": [ { "prefix": "openwhisk", "name": "dockerskeleton", "tag": "latest" } ] }

  CONFIG_whisk_loadbalancer_invokerUserMemory:     2048m
  KAFKA_HOSTS:                                     kafka.openwhisk.svc.cluster.local:9092
  KAFKA_HOST_PORT:                                 9092
  CONFIG_whisk_couchdb_username:                   <set to the key 'db_username' in secret 'db.auth'>        Optional: false
  CONFIG_whisk_couchdb_password:                   <set to the key 'db_password' in secret 'db.auth'>        Optional: false
  CONFIG_whisk_couchdb_port:                       <set to the key 'db_port' of config map 'db.config'>      Optional: false
  CONFIG_whisk_couchdb_protocol:                   <set to the key 'db_protocol' of config map 'db.config'>  Optional: false
  CONFIG_whisk_couchdb_host:                       couchdb.openwhisk.svc.cluster.local
  CONFIG_whisk_couchdb_provider:                   <set to the key 'db_provider' of config map 'db.config'>           Optional: false
  CONFIG_whisk_couchdb_databases_WhiskActivation:  <set to the key 'db_whisk_activations' of config map 'db.config'>  Optional: false
  CONFIG_whisk_couchdb_databases_WhiskEntity:      <set to the key 'db_whisk_actions' of config map 'db.config'>      Optional: false
  CONFIG_whisk_couchdb_databases_WhiskAuth:        <set to the key 'db_whisk_auths' of config map 'db.config'>        Optional: false
  LIMITS_ACTIONS_SEQUENCE_MAXLENGTH:               50
  LIMITS_TRIGGERS_FIRES_PERMINUTE:                 60
  LIMITS_ACTIONS_INVOKES_PERMINUTE:                60
  LIMITS_ACTIONS_INVOKES_CONCURRENT:               30
  CONTROLLER_INSTANCES:                            1
Mounts:
  /var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9qfsx (ro)

Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: ow-core-token-9qfsx: Type: Secret (a volume populated by a Secret) SecretName: ow-core-token-9qfsx Optional: false QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events:

-> kubectl logs controller-0 -n openwhisk Error from server (BadRequest): container "controller" in pod "controller-0" is waiting to start: PodInitializing

-> kubectl -n openwhisk describe pod invoker-b5mpt Name: invoker-b5mpt Namespace: openwhisk Node: server-f2-0f.ipc3dev.com/10.71.3.8 Start Time: Mon, 08 Oct 2018 04:42:25 -0400 Labels: controller-revision-hash=137522596 name=invoker pod-template-generation=1 Annotations: Status: Pending IP: 10.244.1.154 Controlled By: DaemonSet/invoker Init Containers: docker-pull-runtimes: Container ID: docker://83cffcd527aea83fcf4f9f1849cea44c9bc3fe017e9eb73d932032dc77e27a25 Image: openwhisk/ansible-runner:latest Image ID: docker-pullable://docker.io/openwhisk/ansible-runner@sha256:5c9a2354b872a86ac08cba8fee79ba7452c55377f26761bc156247b9026d20d6 Port: Host Port: State: Terminated Reason: Completed Exit Code: 0 Started: Mon, 08 Oct 2018 04:47:05 -0400 Finished: Mon, 08 Oct 2018 05:07:13 -0400 Ready: True Restart Count: 0 Environment: RUNTIMES_MANIFEST: { "runtimes": { "nodejs": [ { "kind": "nodejs", "image": { "prefix": "openwhisk", "name": "nodejsaction", "tag": "latest" }, "deprecated": true }, { "kind": "nodejs:6", "default": true, "image": { "prefix": "openwhisk", "name": "nodejs6action", "tag": "latest" }, "deprecated": false, "stemCells": [{ "count": 2, "memory": "256 MB" }] }, { "kind": "nodejs:8", "default": false, "image": { "prefix": "openwhisk", "name": "action-nodejs-v8", "tag": "latest" }, "deprecated": false } ], "python": [ { "kind": "python", "image": { "prefix": "openwhisk", "name": "python2action", "tag": "latest" }, "deprecated": false }, { "kind": "python:2", "default": true, "image": { "prefix": "openwhisk", "name": "python2action", "tag": "latest" }, "deprecated": false }, { "kind": "python:3", "image": { "prefix": "openwhisk", "name": "python3action", "tag": "latest" }, "deprecated": false } ], "swift": [ { "kind": "swift:3.1.1", "image": { "prefix": "openwhisk", "name": "action-swift-v3.1.1", "tag": "latest" }, "deprecated": false }, { "kind": "swift:4.1", "default": true, "image": { "prefix": "openwhisk", "name": "action-swift-v4.1", "tag": "latest" }, "deprecated": false } ], "java": [ { "kind": "java", "default": true, "image": { "prefix": "openwhisk", "name": "java8action", "tag": "latest" }, "deprecated": false, "attached": { "attachmentName": "jarfile", "attachmentType": "application/java-archive" }, "sentinelledLogs": false, "requireMain": true } ], "php": [ { "kind": "php:7.1", "default": true, "deprecated": false, "image": { "prefix": "openwhisk", "name": "action-php-v7.1", "tag": "latest" } } ], "ruby": [ { "kind": "ruby:2.5", "default": true, "deprecated": false, "image": { "prefix": "openwhisk", "name": "action-ruby-v2.5", "tag": "latest" } } ] }, "blackboxes": [ { "prefix": "openwhisk", "name": "dockerskeleton", "tag": "latest" } ] }

Mounts:
  /task/playbook.yml from task-dir (rw)
  /var/run/docker.sock from dockersock (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from default-token-m4d8z (ro)

wait-for-controller: Container ID: docker://b05867e5c3d1632bbbfa79339f379e43fadc6619e690d24d34fd269fa5de922f Image: busybox Image ID: docker-pullable://docker.io/busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd Port: Host Port: Command: sh -c result=1; until [ $result -eq 0 ]; do echo 'Checking controller readiness'; wget -T 5 --spider $READINESS_URL; result=$?; sleep 1; done; echo 'Success: controller is ready' State: Running Started: Mon, 08 Oct 2018 05:07:14 -0400 Ready: False Restart Count: 0 Environment: READINESS_URL: http://controller.openwhisk.svc.cluster.local:8080/ping Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-m4d8z (ro) Containers: invoker: Container ID: Image: openwhisk/invoker:latest Image ID: Port: 8080/TCP Host Port: 0/TCP Command: /bin/bash -c /init.sh --uniqueName $INVOKER_NAME State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Environment: PORT: 8080 WHISK_API_HOST_PROTO: <set to the key 'whisk_api_host_proto' of config map 'whisk.config'> Optional: false WHISK_API_HOST_PORT: <set to the key 'whisk_api_host_port' of config map 'whisk.config'> Optional: false WHISK_API_HOST_NAME: <set to the key 'whisk_api_host_name' of config map 'whisk.config'> Optional: false INVOKER_CONTAINER_NETWORK: bridge CONFIG_whisk_docker_containerFactory_useRunc: false DOCKER_IMAGE_PREFIX: openwhisk DOCKER_IMAGE_TAG: latest DOCKER_REGISTRY: CONFIG_whisk_loadbalancer_invokerUserMemory: 2048m INVOKER_NAME: (v1:spec.nodeName) JAVA_OPTS: -Xmx512M INVOKER_OPTS: -Dwhisk.spi.ContainerFactoryProvider=whisk.core.containerpool.docker.DockerContainerFactoryProvider RUNTIMES_MANIFEST: { "runtimes": { "nodejs": [ { "kind": "nodejs", "image": { "prefix": "openwhisk", "name": "nodejsaction", "tag": "latest" }, "deprecated": true }, { "kind": "nodejs:6", "default": true, "image": { "prefix": "openwhisk", "name": "nodejs6action", "tag": "latest" }, "deprecated": false, "stemCells": [{ "count": 2, "memory": "256 MB" }] }, { "kind": "nodejs:8", "default": false, "image": { "prefix": "openwhisk", "name": "action-nodejs-v8", "tag": "latest" }, "deprecated": false } ], "python": [ { "kind": "python", "image": { "prefix": "openwhisk", "name": "python2action", "tag": "latest" }, "deprecated": false }, { "kind": "python:2", "default": true, "image": { "prefix": "openwhisk", "name": "python2action", "tag": "latest" }, "deprecated": false }, { "kind": "python:3", "image": { "prefix": "openwhisk", "name": "python3action", "tag": "latest" }, "deprecated": false } ], "swift": [ { "kind": "swift:3.1.1", "image": { "prefix": "openwhisk", "name": "action-swift-v3.1.1", "tag": "latest" }, "deprecated": false }, { "kind": "swift:4.1", "default": true, "image": { "prefix": "openwhisk", "name": "action-swift-v4.1", "tag": "latest" }, "deprecated": false } ], "java": [ { "kind": "java", "default": true, "image": { "prefix": "openwhisk", "name": "java8action", "tag": "latest" }, "deprecated": false, "attached": { "attachmentName": "jarfile", "attachmentType": "application/java-archive" }, "sentinelledLogs": false, "requireMain": true } ], "php": [ { "kind": "php:7.1", "default": true, "deprecated": false, "image": { "prefix": "openwhisk", "name": "action-php-v7.1", "tag": "latest" } } ], "ruby": [ { "kind": "ruby:2.5", "default": true, "deprecated": false, "image": { "prefix": "openwhisk", "name": "action-ruby-v2.5", "tag": "latest" } } ] }, "blackboxes": [ { "prefix": "openwhisk", "name": "dockerskeleton", "tag": "latest" } ] }

  WHISK_LOGS_DIR:
  CONFIG_whisk_info_date:                          <set to the key 'whisk_info_date' of config map 'whisk.config'>  Optional: false
  CONFIG_whisk_couchdb_username:                   <set to the key 'db_username' in secret 'db.auth'>               Optional: false
  CONFIG_whisk_couchdb_password:                   <set to the key 'db_password' in secret 'db.auth'>               Optional: false
  CONFIG_whisk_couchdb_port:                       <set to the key 'db_port' of config map 'db.config'>             Optional: false
  CONFIG_whisk_couchdb_protocol:                   <set to the key 'db_protocol' of config map 'db.config'>         Optional: false
  CONFIG_whisk_couchdb_host:                       couchdb.openwhisk.svc.cluster.local
  CONFIG_whisk_couchdb_provider:                   <set to the key 'db_provider' of config map 'db.config'>           Optional: false
  CONFIG_whisk_couchdb_databases_WhiskActivation:  <set to the key 'db_whisk_activations' of config map 'db.config'>  Optional: false
  CONFIG_whisk_couchdb_databases_WhiskEntity:      <set to the key 'db_whisk_actions' of config map 'db.config'>      Optional: false
  CONFIG_whisk_couchdb_databases_WhiskAuth:        <set to the key 'db_whisk_auths' of config map 'db.config'>        Optional: false
  KAFKA_HOSTS:                                     kafka.openwhisk.svc.cluster.local:9092
  ZOOKEEPER_HOSTS:                                 zookeeper-0.zookeeper.openwhisk.svc.cluster.local:2181
Mounts:
  /containers from dockerrootdir (rw)
  /run/runc from runc (rw)
  /sys/fs/cgroup from cgroup (rw)
  /var/run/docker.sock from dockersock (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from default-token-m4d8z (ro)

Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: cgroup: Type: HostPath (bare host directory volume) Path: /sys/fs/cgroup HostPathType: runc: Type: HostPath (bare host directory volume) Path: /run/runc HostPathType: dockerrootdir: Type: HostPath (bare host directory volume) Path: /var/lib/docker/containers HostPathType: dockersock: Type: HostPath (bare host directory volume) Path: /var/run/docker.sock HostPathType: task-dir: Type: ConfigMap (a volume populated by a ConfigMap) Name: docker-pull-runtimes Optional: false default-token-m4d8z: Type: Secret (a volume populated by a Secret) SecretName: default-token-m4d8z Optional: false QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/disk-pressure:NoSchedule node.kubernetes.io/memory-pressure:NoSchedule node.kubernetes.io/not-ready:NoExecute node.kubernetes.io/unreachable:NoExecute Events:

-> kubectl logs invoker-b5mpt -n openwhisk Error from server (BadRequest): container "invoker" in pod "invoker-b5mpt" is waiting to start: PodInitializing

Please suggest if I am doing something wrong.

dgrove-oss commented 6 years ago

It looks like the controller's wait-for-couchdb init_container is still running (not ready). This is blocking the main controller pod (and thus the main invoker pod).

Try kubectl logs jobs/init-couch-db to see what went wrong with the job that is supposed to initialize the couchDB instance.

saurabh1284 commented 6 years ago

Re-deployed to sync and now Nginx pod is going in CrashLoopBackOff.

-> kubectl get pods -n openwhisk NAME READY STATUS RESTARTS AGE apigateway-64b5fb47df-2c2pf 1/1 Running 0 43m controller-0 0/1 Init:0/2 0 43m couchdb-6db7d989dd-pf8kf 1/1 Running 0 43m install-catalog-8dv5x 0/1 Init:0/1 0 43m install-routemgmt-p2szk 0/1 Init:0/1 0 43m invoker-mpn4x 0/1 Init:1/2 0 43m kafka-0 0/1 Init:0/1 0 43m nginx-55d45778cb-jrtqd 0/1 CrashLoopBackOff 12 43m redis-6788bb4875-vfltv 1/1 Running 0 43m zookeeper-0 1/1 Running 0 43m

-> kubectl describe pod nginx-55d45778cb-jrtqd -n openwhisk Name: nginx-55d45778cb-jrtqd Namespace: openwhisk Node: server-f2-0f.ipc3dev.com/10.71.3.8 Start Time: Tue, 16 Oct 2018 08:16:52 -0400 Labels: name=nginx pod-template-hash=1180133476 Annotations: Status: Running IP: 10.244.1.46 Controlled By: ReplicaSet/nginx-55d45778cb Containers: nginx: Container ID: docker://7ecc08e3c0a1d96cc8bca9be56b5a641f5c2afd67ea82934ac6363cc22edd909 Image: nginx:1.11 Image ID: docker-pullable://docker.io/nginx@sha256:e6693c20186f837fc393390135d8a598a96a833917917789d63766cab6c59582 Ports: 80/TCP, 443/TCP, 8443/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Tue, 16 Oct 2018 08:57:30 -0400 Finished: Tue, 16 Oct 2018 08:57:51 -0400 Ready: False Restart Count: 12 Environment: Mounts: /etc/nginx/certs from nginx-certs (rw) /etc/nginx/nginx.conf from nginx-conf (rw) /logs from logs (rw) /var/run/secrets/kubernetes.io/serviceaccount from default-token-qf7p6 (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: nginx-certs: Type: Secret (a volume populated by a Secret) SecretName: nginx Optional: false nginx-conf: Type: ConfigMap (a volume populated by a ConfigMap) Name: nginx Optional: false logs: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: default-token-qf7p6: Type: Secret (a volume populated by a Secret) SecretName: default-token-qf7p6 Optional: false QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message


Normal Scheduled 44m default-scheduler Successfully assigned openwhisk/nginx-55d45778cb-jrtqd to server-f2-0f.ipc3dev.com Normal Pulled 41m (x5 over 43m) kubelet, server-f2-0f.ipc3dev.com Container image "nginx:1.11" already present on machine Normal Created 41m (x5 over 43m) kubelet, server-f2-0f.ipc3dev.com Created container Normal Started 40m (x5 over 43m) kubelet, server-f2-0f.ipc3dev.com Started container Warning BackOff 3m (x163 over 43m) kubelet, server-f2-0f.ipc3dev.com Back-off restarting failed container

-> kubectl -n openwhisk logs nginx-55d45778cb-jrtqd 2018/10/16 12:57:30 [emerg] 1#1: host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:41 nginx: [emerg] host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:41

-> # kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}

Is it similar to issue #303 ? Do we have any workaround ?

saurabh1284 commented 6 years ago

After latest commit, I am still hitting this issue.. nginx pods are in error with following similar log ==

[root@server-f2-02 incubator-openwhisk-deploy-kube]# kubectl logs nginx-55d45778cb-nmqj6 -n openwhisk 2018/11/02 09:05:30 [emerg] 1#1: host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:41 nginx: [emerg] host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:41

dgrove-oss commented 6 years ago

Did you add an entry to your myCluster.yaml to set a different value for k8s.dns? All the commit did was to make this configurable, but it still defaults to kube-dns.kube-system if you don't override k8s.dns when you deploy the chart.

I would speculate you need something like:

k8s:
  dns: coredns.kube-system
saurabh1284 commented 6 years ago

Tried by adding "coredns.kube-system", but still hitting same issue. -> kubectl logs nginx-55d45778cb-d7ckf -n wsk-k8 2018/11/08 07:51:30 [emerg] 1#1: host not found in resolver "coredns.kube-system" in /etc/nginx/nginx.conf:41 nginx: [emerg] host not found in resolver "coredns.kube-system" in /etc/nginx/nginx.conf:41

I guess we need to get other dns/proxy for nginx, any suggestions ?

dgrove-oss commented 6 years ago

try kubectl get services -n kube-system and see what services are running. It is possible it is named something slightly different (core-dns) for example. I don't have easy access to a cluster that is running coredns, so I am not sure what the service is called.

saurabh1284 commented 6 years ago

thx for inputs. Its working !

Nivekiba commented 4 years ago

@dgrove-oss , sorry for reopen this issue... i try kubectl get services -n kube-system and this is what i got image

There is core-dns nowhere

shiipou commented 4 years ago

I've got the same error :

2020/03/27 07:43:16 [emerg] 1#1: host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:42
nginx: [emerg] host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:42

Here my installation :

As root ($USER is the user I want to set docker group and not env var)

curl -sSL https://get.docker.com/ | sudo bash
sudo usermod -aG docker $USER
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
add-apt-repository "deb https://apt.kubernetes.io/ kubernetes-xenial main"
apt-get update
apt-get install -y kubectl kubeadm kubelet
apt-mark hold kubelet kubeadm kubectl
echo "1" > /proc/sys/net/ipv4/ip_forward
swapoff -a
sed -i -r 's/^(\S+\s+swap\s+.*)$/#\1/' /etc/fstab
kubeadm init --pod-network-cidr=192.168.0.0/16

As $USER

mkdir -p /home/$USER/.kube
cp /etc/kubernetes/admin.conf /home/$USER/.kube/config
chown $USER:$USER /home/$USER/.kube -R
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc6/aio/deploy/recommended.yaml
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

git clone https://github.com/apache/openwhisk-deploy-kube.git
cd git clone openwhisk-deploy-kube

echo "whisk:
  ingress:
    type: NodePort
    apiHostName: 192.168.65.3
    apiHostPort: 31001
nginx:
  httpsNodePort: 31001
k8s:
  dns: kube-dns.kube-system" > custom_values.yml

kubectl create namespace openwhisk
helm install openwhisk ./helm/openwhisk --namespace=openwhisk -f ./custom_values.yml

Here the output of kubectl get services -n kube-system taht I use to set the value of k8s.dns :

NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   42h

I think the helm installation wasn't set the cluster role binding. Without that role, the openwhisk pods can't call other namespaces pods.

I'm wrong ?

shiipou commented 4 years ago

I see here the right YAML to use :

echo "apiVersion: rbac.authorization.k8s.io/v1alpha1
kind: ClusterRoleBinding
metadata:
  name: openwhisk:admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: default
  namespace: openwhisk" | kubectl apply -n openwhisk -f -

But it throw an error to me :

error: unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1alpha1"

to pass the error I need to change the rbac.authorization.k8s.io/v1alpha1 to rbac.authorization.k8s.io/v1beta1