projectodd / openwhisk-openshift

Resources necessary for running OpenWhisk on OpenShift
Apache License 2.0
45 stars 26 forks source link

Can't spin up the Pods in my cluster #35

Closed MohamedElso closed 6 years ago

MohamedElso commented 6 years ago

Hello , i can't spin up the pods for the project in my openshift cluster , here is the Details of Events Tab:

4:05:14 PM Normal Back-off Back-off pulling image "docker-registry ip:5000/openwhiskpoc/busybox:whisky"201 times in the last 47 minutes
3:30:17 PM Warning Failed Error: ImagePullBackOff45 times in the last 47 minutes
3:21:09 PM Warning Failed Error: ErrImagePull3 times in the last 47 minutes
3:21:09 PM Warning Failed Failed to pull image "docker-registry ip:5000/openwhiskpoc/busybox:whisky": rpc error: code = 2 desc = error parsing HTTP 404 response body: json: cannot unmarshal number into Go struct field Error.detail of type float64: "{\"errors\":[{\"code\":\"MANIFEST_UNKNOWN\",\"message\":\"manifest unknown\",\"detail\":{\"Op\":\"Get\",\"URL\":\"https://registry-1.docker.io/v2/\",\… See All3 times in the last 47 minutes

3:21:09 PM | Normal | Pulling | pulling image "docker-registry ip:5000/openwhiskpoc/busybox:whisky"

any suggestions appreciated? Thanks

ryannix123 commented 6 years ago

I think you have to run this, although I don't know how to run it on a non-Minishift cluster. minishift openshift config set --patch \ '{"admissionConfig": {"pluginConfig": {"openshift.io/ImagePolicy": {"configuration": {"apiVersion": "v1", "kind": "ImagePolicyConfig", "resolveImages": "AttemptRewrite"}}}}}'

jcrossley3 commented 6 years ago

Retry with the latest master. I reverted the ImageStream usage last night, so that ImagePolicyConfig patch shouldn't be necessary anymore.

MohamedElso commented 6 years ago

@jcrossley3 thanks, all the pods work good except this pod alarmprovider-1624473681-zctrw , it gives >> Back-off restarting failed container, when I do oc describe pod it doesn't give many details , Do you know the reason ? or how to track it. Thanks


1h 1h 1 default-scheduler Normal Scheduled Successfully assigned alarmprovider-1624473681-zctrw to node ip 1h 1h 1 kubelet, node ip Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "alarmprovider-config" 1h 1h 1 kubelet, node ip Normal SuccessfulMountVolume MountVolume.SetUp succeeded for volume "openwhisk-token-pv2xc" 1h 1h 1 kubelet, node ip spec.initContainers{wait-for-couchdb} Normal Pulling pulling image "busybox" 1h 1h 1 kubelet, node ip spec.initContainers{wait-for-couchdb} Normal Pulled Successfully pulled image "busybox" 1h 1h 1 kubelet, node ip spec.initContainers{wait-for-couchdb} Normal Created Created container 1h 1h 1 kubelet, node ip spec.initContainers{wait-for-couchdb} Normal Started Started container 1h 1h 1 kubelet, node ip spec.initContainers{install-alarms-catalog} Normal Pulling pulling image "projectodd/whisk_alarms:8ee5579" 1h 1h 1 kubelet, node ip spec.initContainers{install-alarms-catalog} Normal Pulled Successfully pulled image "projectodd/whisk_alarms:8ee5579" 1h 1h 5 kubelet, node ip spec.initContainers{install-alarms-catalog} Normal Created Created container 1h 1h 5 kubelet, node ip spec.initContainers{install-alarms-catalog} Normal Started Started container 1h 46m 8 kubelet, node ip spec.initContainers{install-alarms-catalog} Normal Pulled Container image "projectodd/whisk_alarms:8ee5579" already present on machine 1h 1m 246 kubelet, node ip spec.initContainers{install-alarms-catalog} Warning BackOff Back-off restarting failed container

Thanks

jcrossley3 commented 6 years ago

what's the output from oc get pod alarmprovider-1624473681-zctrw -oyaml?

MohamedElso commented 6 years ago

@jcrossley3

apiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/created-by: | {"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"openwhiskpoc","name":"alarmprovider-1624473681","uid":"5ef66909-812d-11e8-8294-005056a70e4e","apiVersion":"extensions","resourceVersion":"53962517"}} kubernetes.io/limit-ranger: 'LimitRanger plugin set: cpu, memory request for container alarmprovider; cpu, memory limit for container alarmprovider; cpu, memory request for init container wait-for-couchdb; cpu, memory limit for init container wait-for-couchdb; cpu, memory request for init container install-alarms-catalog; cpu, memory limit for init container install-alarms-catalog' openshift.io/scc: restricted pod.alpha.kubernetes.io/init-container-statuses: '[{"name":"wait-for-couchdb","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2018-07-06T15:01:13Z","finishedAt":"2018-07-06T15:01:13Z","containerID":"docker://cc218e58389cfd79732d3daeeb31864152e6d86bfe09b3484d68e65500ddc560"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/busybox:latest","imageID":"docker-pullable://docker.io/busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47","containerID":"docker://cc218e58389cfd79732d3daeeb31864152e6d86bfe09b3484d68e65500ddc560"},{"name":"install-alarms-catalog","state":{"waiting":{"reason":"CrashLoopBackOff","message":"Back-off 5m0s restarting failed container=install-alarms-catalog pod=alarmprovider-1624473681-qrkdd_openwhiskpoc(5efe6395-812d-11e8-8294-005056a70e4e)"}},"lastState":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2018-07-06T15:36:40Z","finishedAt":"2018-07-06T15:37:57Z","containerID":"docker://9cae9bd07a5972482ff8e475b43b754ce0a7e1002888673dfd56ba95c00d4464"}},"ready":false,"restartCount":8,"image":"docker.io/projectodd/whisk_alarms:8ee5579","imageID":"docker-pullable://docker.io/projectodd/whisk_alarms@sha256:523105b570612d7d2d5438a92b89f4e6bd86e70e8356dec13633dd3b5e4083b2","containerID":"docker://9cae9bd07a5972482ff8e475b43b754ce0a7e1002888673dfd56ba95c00d4464"}]' pod.alpha.kubernetes.io/init-containers: '[{"name":"wait-for-couchdb","image":"busybox","command":["sh","-cu","echo \"$COUCHDB_SERVICE_HOST\""],"resources":{"limits":{"cpu":"1","memory":"1Gi"},"requests":{"cpu":"100m","memory":"256Mi"}},"volumeMounts":[{"name":"openwhisk-token-xmbhf","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always","securityContext":{"capabilities":{"drop":["KILL","MKNOD","SETGID","SETUID"]},"privileged":false,"seLinuxOptions":{"level":"s0:c27,c4"},"runAsUser":1000710000}},{"name":"install-alarms-catalog","image":"projectodd/whisk_alarms:8ee5579","envFrom":[{"configMapRef":{"name":"alarmprovider"}}],"env":[{"name":"DB_HOST","value":"$(COUCHDB_SERVICE_HOST):$(COUCHDB_SERVICE_PORT_COUCHDB)"},{"name":"DB_PROTOCOL","valueFrom":{"configMapKeyRef":{"name":"db.config","key":"db_protocol"}}},{"name":"AUTH_WHISK_SYSTEM","valueFrom":{"secretKeyRef":{"name":"whisk.auth","key":"system"}}}],"resources":{"limits":{"cpu":"1","memory":"1Gi"},"requests":{"cpu":"100m","memory":"256Mi"}},"volumeMounts":[{"name":"alarmprovider-config","mountPath":"/alarmprovider_config"},{"name":"openwhisk-token-xmbhf","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"drop":["KILL","MKNOD","SETGID","SETUID"]},"privileged":false,"seLinuxOptions":{"level":"s0:c27,c4"},"runAsUser":1000710000}}]' pod.beta.kubernetes.io/init-container-statuses: '[{"name":"wait-for-couchdb","state":{"terminated":{"exitCode":0,"reason":"Completed","startedAt":"2018-07-06T15:01:13Z","finishedAt":"2018-07-06T15:01:13Z","containerID":"docker://cc218e58389cfd79732d3daeeb31864152e6d86bfe09b3484d68e65500ddc560"}},"lastState":{},"ready":true,"restartCount":0,"image":"docker.io/busybox:latest","imageID":"docker-pullable://docker.io/busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47","containerID":"docker://cc218e58389cfd79732d3daeeb31864152e6d86bfe09b3484d68e65500ddc560"},{"name":"install-alarms-catalog","state":{"waiting":{"reason":"CrashLoopBackOff","message":"Back-off 5m0s restarting failed container=install-alarms-catalog pod=alarmprovider-1624473681-qrkdd_openwhiskpoc(5efe6395-812d-11e8-8294-005056a70e4e)"}},"lastState":{"terminated":{"exitCode":1,"reason":"Error","startedAt":"2018-07-06T15:36:40Z","finishedAt":"2018-07-06T15:37:57Z","containerID":"docker://9cae9bd07a5972482ff8e475b43b754ce0a7e1002888673dfd56ba95c00d4464"}},"ready":false,"restartCount":8,"image":"docker.io/projectodd/whisk_alarms:8ee5579","imageID":"docker-pullable://docker.io/projectodd/whisk_alarms@sha256:523105b570612d7d2d5438a92b89f4e6bd86e70e8356dec13633dd3b5e4083b2","containerID":"docker://9cae9bd07a5972482ff8e475b43b754ce0a7e1002888673dfd56ba95c00d4464"}]' pod.beta.kubernetes.io/init-containers: '[{"name":"wait-for-couchdb","image":"busybox","command":["sh","-cu","echo \"$COUCHDB_SERVICE_HOST\""],"resources":{"limits":{"cpu":"1","memory":"1Gi"},"requests":{"cpu":"100m","memory":"256Mi"}},"volumeMounts":[{"name":"openwhisk-token-xmbhf","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"Always","securityContext":{"capabilities":{"drop":["KILL","MKNOD","SETGID","SETUID"]},"privileged":false,"seLinuxOptions":{"level":"s0:c27,c4"},"runAsUser":1000710000}},{"name":"install-alarms-catalog","image":"projectodd/whisk_alarms:8ee5579","envFrom":[{"configMapRef":{"name":"alarmprovider"}}],"env":[{"name":"DB_HOST","value":"$(COUCHDB_SERVICE_HOST):$(COUCHDB_SERVICE_PORT_COUCHDB)"},{"name":"DB_PROTOCOL","valueFrom":{"configMapKeyRef":{"name":"db.config","key":"db_protocol"}}},{"name":"AUTH_WHISK_SYSTEM","valueFrom":{"secretKeyRef":{"name":"whisk.auth","key":"system"}}}],"resources":{"limits":{"cpu":"1","memory":"1Gi"},"requests":{"cpu":"100m","memory":"256Mi"}},"volumeMounts":[{"name":"alarmprovider-config","mountPath":"/alarmprovider_config"},{"name":"openwhisk-token-xmbhf","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"drop":["KILL","MKNOD","SETGID","SETUID"]},"privileged":false,"seLinuxOptions":{"level":"s0:c27,c4"},"runAsUser":1000710000}}]' creationTimestamp: 2018-07-06T15:00:50Z generateName: alarmprovider-1624473681- labels: name: alarmprovider pod-template-hash: "1624473681" name: alarmprovider-1624473681-qrkdd namespace: openwhiskpoc ownerReferences:

Thanks

MohamedElso commented 6 years ago

@jcrossley3 and i have this status on the pod alarmprovider-1624473681-qrkdd 0/1 Init:CrashLoopBackOff 11 58m

Thanks for your help!

jcrossley3 commented 6 years ago

So that's clearly showing an error with the install-alarms-catalog init container.

Try oc logs alarmprovider-1624473681-zctrw -c install-alarms-catalog

MohamedElso commented 6 years ago

@jcrossley3 Waiting for controller to be available

jcrossley3 commented 6 years ago

I have no idea why npm would be giving that error. I see no such error on minishift. :(

MohamedElso commented 6 years ago

@jcrossley3 I was thinking about putting something " NO_PROXY= "section in docker file on master so we can skip downloading the package problem, or add some parameters to the template when deploying,or add some parameters to docker images files what do u think? @jcrossley3 where this directory exist , npm ERR! /openwhisk-package-alarms/action/npm-debug.log ? look at this https://github.com/npm/npm/issues/8872 Thanks

MohamedElso commented 6 years ago

@jcrossley3 any updates/suggestions on how to bypass this error ... thanks

jcrossley3 commented 6 years ago

This is on a clean, recent minishift with the latest master?

MohamedElso commented 6 years ago

@jcrossley3 yes i used the template that you updated 6 days ago , all goes well except the error in spinning up the pod "alarmprovider" related to npm ERR as i described in recent posts . @bbrowning i am using cluster openshift with this info: Version OpenShift Master: v3.7.44 Kubernetes Master: v1.7.6+a08f5eeb62 Thanks