Closed saurabh1284 closed 6 years ago
It looks like the controller's wait-for-couchdb
init_container is still running (not ready). This is blocking the main controller pod (and thus the main invoker pod).
Try kubectl logs jobs/init-couch-db
to see what went wrong with the job that is supposed to initialize the couchDB instance.
Re-deployed to sync and now Nginx pod is going in CrashLoopBackOff.
-> kubectl get pods -n openwhisk NAME READY STATUS RESTARTS AGE apigateway-64b5fb47df-2c2pf 1/1 Running 0 43m controller-0 0/1 Init:0/2 0 43m couchdb-6db7d989dd-pf8kf 1/1 Running 0 43m install-catalog-8dv5x 0/1 Init:0/1 0 43m install-routemgmt-p2szk 0/1 Init:0/1 0 43m invoker-mpn4x 0/1 Init:1/2 0 43m kafka-0 0/1 Init:0/1 0 43m nginx-55d45778cb-jrtqd 0/1 CrashLoopBackOff 12 43m redis-6788bb4875-vfltv 1/1 Running 0 43m zookeeper-0 1/1 Running 0 43m
-> kubectl describe pod nginx-55d45778cb-jrtqd -n openwhisk
Name: nginx-55d45778cb-jrtqd
Namespace: openwhisk
Node: server-f2-0f.ipc3dev.com/10.71.3.8
Start Time: Tue, 16 Oct 2018 08:16:52 -0400
Labels: name=nginx
pod-template-hash=1180133476
Annotations:
Normal Scheduled 44m default-scheduler Successfully assigned openwhisk/nginx-55d45778cb-jrtqd to server-f2-0f.ipc3dev.com Normal Pulled 41m (x5 over 43m) kubelet, server-f2-0f.ipc3dev.com Container image "nginx:1.11" already present on machine Normal Created 41m (x5 over 43m) kubelet, server-f2-0f.ipc3dev.com Created container Normal Started 40m (x5 over 43m) kubelet, server-f2-0f.ipc3dev.com Started container Warning BackOff 3m (x163 over 43m) kubelet, server-f2-0f.ipc3dev.com Back-off restarting failed container
-> kubectl -n openwhisk logs nginx-55d45778cb-jrtqd 2018/10/16 12:57:30 [emerg] 1#1: host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:41 nginx: [emerg] host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:41
-> # kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:08:34Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Is it similar to issue #303 ? Do we have any workaround ?
After latest commit, I am still hitting this issue.. nginx pods are in error with following similar log ==
[root@server-f2-02 incubator-openwhisk-deploy-kube]# kubectl logs nginx-55d45778cb-nmqj6 -n openwhisk 2018/11/02 09:05:30 [emerg] 1#1: host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:41 nginx: [emerg] host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:41
Did you add an entry to your myCluster.yaml
to set a different value for k8s.dns
? All the commit did was to make this configurable, but it still defaults to kube-dns.kube-system
if you don't override k8s.dns
when you deploy the chart.
I would speculate you need something like:
k8s:
dns: coredns.kube-system
Tried by adding "coredns.kube-system", but still hitting same issue. -> kubectl logs nginx-55d45778cb-d7ckf -n wsk-k8 2018/11/08 07:51:30 [emerg] 1#1: host not found in resolver "coredns.kube-system" in /etc/nginx/nginx.conf:41 nginx: [emerg] host not found in resolver "coredns.kube-system" in /etc/nginx/nginx.conf:41
I guess we need to get other dns/proxy for nginx, any suggestions ?
try kubectl get services -n kube-system
and see what services are running. It is possible it is named something slightly different (core-dns
) for example. I don't have easy access to a cluster that is running coredns, so I am not sure what the service is called.
thx for inputs. Its working !
@dgrove-oss , sorry for reopen this issue... i try kubectl get services -n kube-system
and this is what i got
There is core-dns
nowhere
I've got the same error :
2020/03/27 07:43:16 [emerg] 1#1: host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:42
nginx: [emerg] host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:42
Here my installation :
As root ($USER is the user I want to set docker group and not env var)
curl -sSL https://get.docker.com/ | sudo bash
sudo usermod -aG docker $USER
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
add-apt-repository "deb https://apt.kubernetes.io/ kubernetes-xenial main"
apt-get update
apt-get install -y kubectl kubeadm kubelet
apt-mark hold kubelet kubeadm kubectl
echo "1" > /proc/sys/net/ipv4/ip_forward
swapoff -a
sed -i -r 's/^(\S+\s+swap\s+.*)$/#\1/' /etc/fstab
kubeadm init --pod-network-cidr=192.168.0.0/16
As $USER
mkdir -p /home/$USER/.kube
cp /etc/kubernetes/admin.conf /home/$USER/.kube/config
chown $USER:$USER /home/$USER/.kube -R
kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc6/aio/deploy/recommended.yaml
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml
git clone https://github.com/apache/openwhisk-deploy-kube.git
cd git clone openwhisk-deploy-kube
echo "whisk:
ingress:
type: NodePort
apiHostName: 192.168.65.3
apiHostPort: 31001
nginx:
httpsNodePort: 31001
k8s:
dns: kube-dns.kube-system" > custom_values.yml
kubectl create namespace openwhisk
helm install openwhisk ./helm/openwhisk --namespace=openwhisk -f ./custom_values.yml
Here the output of kubectl get services -n kube-system
taht I use to set the value of k8s.dns
:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 42h
I think the helm installation wasn't set the cluster role binding. Without that role, the openwhisk pods can't call other namespaces pods.
I'm wrong ?
I see here the right YAML to use :
echo "apiVersion: rbac.authorization.k8s.io/v1alpha1
kind: ClusterRoleBinding
metadata:
name: openwhisk:admin
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: default
namespace: openwhisk" | kubectl apply -n openwhisk -f -
But it throw an error to me :
error: unable to recognize "STDIN": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1alpha1"
to pass the error I need to change the rbac.authorization.k8s.io/v1alpha1
to rbac.authorization.k8s.io/v1beta1
I have an kubernetes cluster of 2 nodes (1 master and 1 worker) and was trying to deploy openwhisk following documentation steps. But controller and invoker pods are stuck in PodInitializing phase. I am using latest clone. Please find details as below:
-> kubectl get nodes --show-labels NAME STATUS ROLES AGE VERSION LABELS server-f2-02.ipc3dev.com Ready master 33d v1.11.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=server-f2-02.ipc3dev.com,node-role.kubernetes.io/master=master server-f2-0f.ipc3dev.com Ready worker 33d v1.11.0 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=server-f2-0f.ipc3dev.com,node-role.kubernetes.io/worker=worker,openwhisk-role=invoker
-> kubectl describe nodes | grep InternalIP nternalIP: 10.71.3.7 InternalIP: 10.71.3.8
-> cat mycluster.yaml whisk: ingress: type: NodePort api_host_name: 10.71.3.7 api_host_port: 31001 nginx: httpsNodePort: 31001
-> kubectl -n openwhisk get pods NAME READY STATUS RESTARTS AGE apigateway-64b5fb47df-h8v5f 1/1 Running 0 2h controller-0 0/1 Init:1/2 0 2h couchdb-6db7d989dd-886mc 1/1 Running 0 2h invoker-b5mpt 0/1 Init:1/2 0 2h kafka-0 1/1 Running 0 2h nginx-55d45778cb-5vgd6 1/1 Running 0 2h redis-6788bb4875-lcwgw 1/1 Running 0 2h zookeeper-0 1/1 Running 0 2h
-> kubectl -n openwhisk describe pod controller-0 Name: controller-0 Namespace: openwhisk Node: server-f2-0f.ipc3dev.com/10.71.3.8 Start Time: Mon, 08 Oct 2018 04:42:25 -0400 Labels: controller-revision-hash=controller-5bc6758f68 name=controller statefulset.kubernetes.io/pod-name=controller-0 Annotations:
Status: Pending
IP: 10.244.1.156
Controlled By: StatefulSet/controller
Init Containers:
wait-for-kafka:
Container ID: docker://6d870e1e10ab5913fd8f48c00f7ee342c7e8e9e0af25a727a15d2229ccb4d00a
Image: busybox
Image ID: docker-pullable://docker.io/busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd
Port:
Host Port:
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 zookeeper-0.zookeeper.openwhisk.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi done; echo "Zookeeper is up; will wait for 10 seconds to give kafka time to initialize"; sleep 10;
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 08 Oct 2018 04:42:37 -0400
Finished: Mon, 08 Oct 2018 04:43:17 -0400
Ready: True
Restart Count: 0
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9qfsx (ro)
wait-for-couchdb:
Container ID: docker://08391dc6b3eb76127050c35f0948006a0e398d7b761dc208f83c2b621f4a5c64
Image: busybox
Image ID: docker-pullable://docker.io/busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd
Port:
Host Port:
Command:
sh
-c
while true; do echo 'checking CouchDB readiness'; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; if [ $result -eq 0 ]; then echo 'Success: CouchDB is ready!'; break; fi; echo '...not ready yet; sleeping 3 seconds before retry'; sleep 3; done;
State: Running
Started: Mon, 08 Oct 2018 04:43:19 -0400
Ready: False
Restart Count: 0
Environment:
READINESS_URL: http://couchdb.openwhisk.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-9qfsx (ro)
Containers:
controller:
Container ID:
Image: openwhisk/controller:latest
Image ID:
Ports: 8080/TCP, 2552/TCP, 19999/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Command:
/bin/bash
-c
/init.sh
hostname | cut -d'-' -f2
State: Waiting Reason: PodInitializing Ready: False Restart Count: 0 Liveness: http-get http://:8080/ping delay=5s timeout=1s period=10s #success=1 #failure=3 Environment: PORT: 8080 CONFIG_whisk_info_date: <set to the key 'whisk_info_date' of config map 'whisk.config'> Optional: false CONFIG_whisk_info_buildNo: <set to the key 'whisk_info_buildNo' of config map 'whisk.config'> Optional: false JAVA_OPTS: -Xmx1024M CONTROLLER_OPTS: RUNTIMES_MANIFEST: { "runtimes": { "nodejs": [ { "kind": "nodejs", "image": { "prefix": "openwhisk", "name": "nodejsaction", "tag": "latest" }, "deprecated": true }, { "kind": "nodejs:6", "default": true, "image": { "prefix": "openwhisk", "name": "nodejs6action", "tag": "latest" }, "deprecated": false, "stemCells": [{ "count": 2, "memory": "256 MB" }] }, { "kind": "nodejs:8", "default": false, "image": { "prefix": "openwhisk", "name": "action-nodejs-v8", "tag": "latest" }, "deprecated": false } ], "python": [ { "kind": "python", "image": { "prefix": "openwhisk", "name": "python2action", "tag": "latest" }, "deprecated": false }, { "kind": "python:2", "default": true, "image": { "prefix": "openwhisk", "name": "python2action", "tag": "latest" }, "deprecated": false }, { "kind": "python:3", "image": { "prefix": "openwhisk", "name": "python3action", "tag": "latest" }, "deprecated": false } ], "swift": [ { "kind": "swift:3.1.1", "image": { "prefix": "openwhisk", "name": "action-swift-v3.1.1", "tag": "latest" }, "deprecated": false }, { "kind": "swift:4.1", "default": true, "image": { "prefix": "openwhisk", "name": "action-swift-v4.1", "tag": "latest" }, "deprecated": false } ], "java": [ { "kind": "java", "default": true, "image": { "prefix": "openwhisk", "name": "java8action", "tag": "latest" }, "deprecated": false, "attached": { "attachmentName": "jarfile", "attachmentType": "application/java-archive" }, "sentinelledLogs": false, "requireMain": true } ], "php": [ { "kind": "php:7.1", "default": true, "deprecated": false, "image": { "prefix": "openwhisk", "name": "action-php-v7.1", "tag": "latest" } } ], "ruby": [ { "kind": "ruby:2.5", "default": true, "deprecated": false, "image": { "prefix": "openwhisk", "name": "action-ruby-v2.5", "tag": "latest" } } ] }, "blackboxes": [ { "prefix": "openwhisk", "name": "dockerskeleton", "tag": "latest" } ] }Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: ow-core-token-9qfsx: Type: Secret (a volume populated by a Secret) SecretName: ow-core-token-9qfsx Optional: false QoS Class: BestEffort Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
-> kubectl logs controller-0 -n openwhisk Error from server (BadRequest): container "controller" in pod "controller-0" is waiting to start: PodInitializing
-> kubectl -n openwhisk describe pod invoker-b5mpt Name: invoker-b5mpt Namespace: openwhisk Node: server-f2-0f.ipc3dev.com/10.71.3.8 Start Time: Mon, 08 Oct 2018 04:42:25 -0400 Labels: controller-revision-hash=137522596 name=invoker pod-template-generation=1 Annotations:
Status: Pending
IP: 10.244.1.154
Controlled By: DaemonSet/invoker
Init Containers:
docker-pull-runtimes:
Container ID: docker://83cffcd527aea83fcf4f9f1849cea44c9bc3fe017e9eb73d932032dc77e27a25
Image: openwhisk/ansible-runner:latest
Image ID: docker-pullable://docker.io/openwhisk/ansible-runner@sha256:5c9a2354b872a86ac08cba8fee79ba7452c55377f26761bc156247b9026d20d6
Port:
Host Port:
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 08 Oct 2018 04:47:05 -0400
Finished: Mon, 08 Oct 2018 05:07:13 -0400
Ready: True
Restart Count: 0
Environment:
RUNTIMES_MANIFEST: {
"runtimes": {
"nodejs": [
{
"kind": "nodejs",
"image": {
"prefix": "openwhisk",
"name": "nodejsaction",
"tag": "latest"
},
"deprecated": true
},
{
"kind": "nodejs:6",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "nodejs6action",
"tag": "latest"
},
"deprecated": false,
"stemCells": [{
"count": 2,
"memory": "256 MB"
}]
},
{
"kind": "nodejs:8",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v8",
"tag": "latest"
},
"deprecated": false
}
],
"python": [
{
"kind": "python",
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:2",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:3",
"image": {
"prefix": "openwhisk",
"name": "python3action",
"tag": "latest"
},
"deprecated": false
}
],
"swift": [
{
"kind": "swift:3.1.1",
"image": {
"prefix": "openwhisk",
"name": "action-swift-v3.1.1",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "swift:4.1",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v4.1",
"tag": "latest"
},
"deprecated": false
}
],
"java": [
{
"kind": "java",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "java8action",
"tag": "latest"
},
"deprecated": false,
"attached": {
"attachmentName": "jarfile",
"attachmentType": "application/java-archive"
},
"sentinelledLogs": false,
"requireMain": true
}
],
"php": [
{
"kind": "php:7.1",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.1",
"tag": "latest"
}
}
],
"ruby": [
{
"kind": "ruby:2.5",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-ruby-v2.5",
"tag": "latest"
}
}
]
},
"blackboxes": [
{
"prefix": "openwhisk",
"name": "dockerskeleton",
"tag": "latest"
}
]
}
wait-for-controller: Container ID: docker://b05867e5c3d1632bbbfa79339f379e43fadc6619e690d24d34fd269fa5de922f Image: busybox Image ID: docker-pullable://docker.io/busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd Port:
Host Port:
Command:
sh
-c
result=1; until [ $result -eq 0 ]; do echo 'Checking controller readiness'; wget -T 5 --spider $READINESS_URL; result=$?; sleep 1; done; echo 'Success: controller is ready'
State: Running
Started: Mon, 08 Oct 2018 05:07:14 -0400
Ready: False
Restart Count: 0
Environment:
READINESS_URL: http://controller.openwhisk.svc.cluster.local:8080/ping
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-m4d8z (ro)
Containers:
invoker:
Container ID:
Image: openwhisk/invoker:latest
Image ID:
Port: 8080/TCP
Host Port: 0/TCP
Command:
/bin/bash
-c
/init.sh --uniqueName $INVOKER_NAME
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
PORT: 8080
WHISK_API_HOST_PROTO: <set to the key 'whisk_api_host_proto' of config map 'whisk.config'> Optional: false
WHISK_API_HOST_PORT: <set to the key 'whisk_api_host_port' of config map 'whisk.config'> Optional: false
WHISK_API_HOST_NAME: <set to the key 'whisk_api_host_name' of config map 'whisk.config'> Optional: false
INVOKER_CONTAINER_NETWORK: bridge
CONFIG_whisk_docker_containerFactory_useRunc: false
DOCKER_IMAGE_PREFIX: openwhisk
DOCKER_IMAGE_TAG: latest
DOCKER_REGISTRY:
CONFIG_whisk_loadbalancer_invokerUserMemory: 2048m
INVOKER_NAME: (v1:spec.nodeName)
JAVA_OPTS: -Xmx512M
INVOKER_OPTS: -Dwhisk.spi.ContainerFactoryProvider=whisk.core.containerpool.docker.DockerContainerFactoryProvider
RUNTIMES_MANIFEST: {
"runtimes": {
"nodejs": [
{
"kind": "nodejs",
"image": {
"prefix": "openwhisk",
"name": "nodejsaction",
"tag": "latest"
},
"deprecated": true
},
{
"kind": "nodejs:6",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "nodejs6action",
"tag": "latest"
},
"deprecated": false,
"stemCells": [{
"count": 2,
"memory": "256 MB"
}]
},
{
"kind": "nodejs:8",
"default": false,
"image": {
"prefix": "openwhisk",
"name": "action-nodejs-v8",
"tag": "latest"
},
"deprecated": false
}
],
"python": [
{
"kind": "python",
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:2",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "python2action",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "python:3",
"image": {
"prefix": "openwhisk",
"name": "python3action",
"tag": "latest"
},
"deprecated": false
}
],
"swift": [
{
"kind": "swift:3.1.1",
"image": {
"prefix": "openwhisk",
"name": "action-swift-v3.1.1",
"tag": "latest"
},
"deprecated": false
},
{
"kind": "swift:4.1",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "action-swift-v4.1",
"tag": "latest"
},
"deprecated": false
}
],
"java": [
{
"kind": "java",
"default": true,
"image": {
"prefix": "openwhisk",
"name": "java8action",
"tag": "latest"
},
"deprecated": false,
"attached": {
"attachmentName": "jarfile",
"attachmentType": "application/java-archive"
},
"sentinelledLogs": false,
"requireMain": true
}
],
"php": [
{
"kind": "php:7.1",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-php-v7.1",
"tag": "latest"
}
}
],
"ruby": [
{
"kind": "ruby:2.5",
"default": true,
"deprecated": false,
"image": {
"prefix": "openwhisk",
"name": "action-ruby-v2.5",
"tag": "latest"
}
}
]
},
"blackboxes": [
{
"prefix": "openwhisk",
"name": "dockerskeleton",
"tag": "latest"
}
]
}
Conditions: Type Status Initialized False Ready False ContainersReady False PodScheduled True Volumes: cgroup: Type: HostPath (bare host directory volume) Path: /sys/fs/cgroup HostPathType: runc: Type: HostPath (bare host directory volume) Path: /run/runc HostPathType: dockerrootdir: Type: HostPath (bare host directory volume) Path: /var/lib/docker/containers HostPathType: dockersock: Type: HostPath (bare host directory volume) Path: /var/run/docker.sock HostPathType: task-dir: Type: ConfigMap (a volume populated by a ConfigMap) Name: docker-pull-runtimes Optional: false default-token-m4d8z: Type: Secret (a volume populated by a Secret) SecretName: default-token-m4d8z Optional: false QoS Class: BestEffort Node-Selectors:
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/unreachable:NoExecute
Events:
-> kubectl logs invoker-b5mpt -n openwhisk Error from server (BadRequest): container "invoker" in pod "invoker-b5mpt" is waiting to start: PodInitializing
Please suggest if I am doing something wrong.