search

apache / openwhisk-deploy-kube

The Apache OpenWhisk Kubernetes Deployment repository supports deploying the Apache OpenWhisk system on Kubernetes and OpenShift clusters.
https://openwhisk.apache.org/
Apache License 2.0
300 stars 231 forks source link

Errors on pod/install-catalog, pod/routemgmt #336

Closed JiniousChoi closed 5 years ago

JiniousChoi commented 5 years ago 
jinc@jinc-MacBookPro:~/github$ kubectl get all -n openwhisk -o wide
NAME                              READY     STATUS      RESTARTS   AGE       IP            NODE
pod/apigateway-7b87dd957f-ng29r   1/1       Running     0          6m        10.244.2.5    kube-node-1
pod/controller-0                  1/1       Running     0          6m        10.244.2.4    kube-node-1
pod/couchdb-6b87ccfb78-hvqjv      1/1       Running     0          6m        10.244.3.4    kube-node-2
pod/init-couchdb-gn7b7            0/1       Completed   0          6m        10.244.2.8    kube-node-1
pod/install-catalog-8x7qq         0/1       Error       0          6m        10.244.3.7    kube-node-2
pod/install-catalog-bnj8f         0/1       Error       0          58s       10.244.3.9    kube-node-2
pod/install-catalog-d7c6j         0/1       Error       0          38s       10.244.3.10   kube-node-2
pod/install-catalog-s22cm         0/1       Error       0          1m        10.244.2.10   kube-node-1
pod/install-routemgmt-dhfzr       0/1       Error       0          47s       10.244.2.11   kube-node-1
pod/install-routemgmt-szlf8       0/1       Error       0          6m        10.244.2.7    kube-node-1
pod/install-routemgmt-vh5lp       1/1       Running     0          5s        10.244.2.12   kube-node-1
pod/invoker-r48jz                 1/1       Running     0          6m        10.244.2.3    kube-node-1
pod/invoker-sq8nr                 1/1       Running     0          6m        10.244.3.6    kube-node-2
pod/kafka-0                       1/1       Running     0          6m        10.244.2.9    kube-node-1
pod/nginx-598c75d7d6-6c5sl        1/1       Running     0          6m        10.244.2.6    kube-node-1
pod/redis-5d77674f65-csl6d        1/1       Running     0          6m        10.244.3.5    kube-node-2
pod/zookeeper-0                   1/1       Running     0          6m        10.244.3.8    kube-node-2

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                                     AGE       SELECTOR
service/apigateway   ClusterIP   10.104.72.202    <none>        8080/TCP,9000/TCP                           6m        name=apigateway
service/controller   ClusterIP   10.102.29.83     <none>        8080/TCP                                    6m        name=controller
service/couchdb      ClusterIP   10.107.45.222    <none>        5984/TCP                                    6m        name=couchdb
service/kafka        ClusterIP   10.111.233.124   <none>        9092/TCP                                    6m        name=kafka
service/nginx        NodePort    10.105.80.76     <none>        80:31520/TCP,443:31001/TCP,8443:30304/TCP   6m        name=nginx
service/redis        ClusterIP   10.103.8.64      <none>        6379/TCP                                    6m        name=redis
service/zookeeper    ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP                  6m        name=zookeeper

NAME                     DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE       CONTAINERS   IMAGES                     SELECTOR
daemonset.apps/invoker   2         2         2         2            2           <none>          6m        invoker      openwhisk/invoker:latest   name=invoker

NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINERS   IMAGES                        SELECTOR
deployment.apps/apigateway   1         1         1            1           6m        apigateway   openwhisk/apigateway:latest   name=apigateway
deployment.apps/couchdb      1         1         1            1           6m        couchdb      apache/couchdb:2.1            name=couchdb
deployment.apps/nginx        1         1         1            1           6m        nginx        nginx:1.11                    name=nginx
deployment.apps/redis        1         1         1            1           6m        redis        redis:3.2                     name=redis

NAME                                    DESIRED   CURRENT   READY     AGE       CONTAINERS   IMAGES                        SELECTOR
replicaset.apps/apigateway-7b87dd957f   1         1         1         6m        apigateway   openwhisk/apigateway:latest   name=apigateway,pod-template-hash=3643885139
replicaset.apps/couchdb-6b87ccfb78      1         1         1         6m        couchdb      apache/couchdb:2.1            name=couchdb,pod-template-hash=2643779634
replicaset.apps/nginx-598c75d7d6        1         1         1         6m        nginx        nginx:1.11                    name=nginx,pod-template-hash=1547318382
replicaset.apps/redis-5d77674f65        1         1         1         6m        redis        redis:3.2                     name=redis,pod-template-hash=1833230921

NAME                          DESIRED   CURRENT   AGE       CONTAINERS   IMAGES
statefulset.apps/controller   1         1         6m        controller   openwhisk/controller:latest
statefulset.apps/kafka        1         1         6m        kafka        wurstmeister/kafka:0.11.0.1
statefulset.apps/zookeeper    1         1         6m        zookeeper    zookeeper:3.4

NAME                          DESIRED   SUCCESSFUL   AGE       CONTAINERS     IMAGES                            SELECTOR
job.batch/init-couchdb        1         1            6m        init-couchdb   openwhisk/ansible-runner:latest   controller-uid=11e7da9c-e0cb-11e8-a6b0-363c7332d8e7
job.batch/install-catalog     1         0            6m        catalog        openwhisk/script-runner:latest    controller-uid=11e96ab1-e0cb-11e8-a6b0-363c7332d8e7
job.batch/install-routemgmt   1         0            6m        routemgmt      openwhisk/script-runner:latest    controller-uid=11e9ecbf-e0cb-11e8-a6b0-363c7332d8e7

logs for pod/install-catalog-s22cm

jinc@jinc-MacBookPro:~/github$ kubectl logs pod/install-catalog-s22cm -n openwhisk
+ '[' -f /task/myTask.sh ']'
+ . /task/myTask.sh
++ export OPENWHISK_HOME=/openwhisk
++ OPENWHISK_HOME=/openwhisk
++ git clone https://github.com/apache/incubator-openwhisk-catalog openwhisk-catalog
Cloning into 'openwhisk-catalog'...
++ pushd openwhisk-catalog/packages
++ ./installCatalog.sh 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP nil:nil /usr/local/bin/wsk
/openwhisk-catalog/packages /
Installing OpenWhisk packages
Installing package installCombinators.sh with pid 24
Installing package installGit.sh with pid 25
Installing package installSlack.sh with pid 26
Installing package installSystem.sh with pid 27
Installing package installWatson.sh with pid 28
Installing package installWeather.sh with pid 29
Installing package installWebSocket.sh with pid 30
Installing Git package.
Creating package github with pid 54
Installing Slack package.
Installing action combinator package.
Creating package combinators with pid 71
Creating package slack with pid 72
Installing WebSocket package.
Installing Weather package.
Creating package websocket with pid 87
Creating package weather with pid 89
Installing whisk.system entities.
Creating package utils with pid 91
Installing Watson package.
Creating package samples with pid 92
Creating package watson-speechToText with pid 93
Creating package watson-textToSpeech with pid 97
error: Package update failed: Put https://nil:nil/api/v1/namespaces/_/packages/github?overwrite=true: invalid URL port "nil"
54 finished with status 3
Installing github/webhook with pid 120
error: Package update failed: Put https://nil:nil/api/v1/namespaces/_/packages/combinators?overwrite=true: invalid URL port "nil"
error: Package update failed: Put https://nil:nil/api/v1/namespaces/_/packages/websocket?overwrite=true: invalid URL port "nil"
87 finished with status 3
71 finished with status 3
Installing websocket/send with pid 121
Installing combinators/eca with pid 122
Installing combinators/forwarder with pid 123
Installing combinators/retry with pid 124
error: Package update failed: Put https://nil:nil/api/v1/namespaces/_/packages/slack?overwrite=true: invalid URL port "nil"
Installing combinators/trycatch with pid 125
72 finished with status 3
Installing slack/post with pid 141
error: Package update failed: Put https://nil:nil/api/v1/namespaces/_/packages/samples?overwrite=true: invalid URL port "nil"
error: Package update failed: Put https://nil:nil/api/v1/namespaces/_/packages/weather?overwrite=true: invalid URL port "nil"
89 finished with status 3
Installing weather/forecast with pid 154
error: Package update failed: Put https://nil:nil/api/v1/namespaces/_/packages/utils?overwrite=true: invalid URL port "nil"
91 finished with status 3
92 finished with status 3
Installing utils/echo with pid 155
Installing utils/cat with pid 156
Installing utils/smash with pid 160
error: Package update failed: Put https://nil:nil/api/v1/namespaces/_/packages/watson-speechToText?overwrite=true: invalid URL port "nil"
Installing utils/split with pid 161
93 finished with status 3
Installing utils/sort with pid 166
Installing utils/head with pid 167
Installing utils/date with pid 169
Installing utils/namespace with pid 172
Installing utils/hosturl with pid 176
error: Package update failed: Put https://nil:nil/api/v1/namespaces/_/packages/watson-textToSpeech?overwrite=true: invalid URL port "nil"
Installing samples/helloWorld with pid 180
Installing samples/greeting with pid 181
Installing samples/wordCount with pid 185
Installing samples/curl with pid 189
97 finished with status 3
error: Invalid argument(s). Unable to get action 'combinators/trycatch': Get https://nil:nil/api/v1/namespaces/_/actions/combinators/trycatch?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
Installing watson-speechToText/speechToText with pid 209
error: Invalid argument(s). Unable to get action 'combinators/retry': Get https://nil:nil/api/v1/namespaces/_/actions/combinators/retry?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
Installing watson-textToSpeech/textToSpeech with pid 214
error: Invalid argument(s). Unable to get action 'github/webhook': Get https://nil:nil/api/v1/namespaces/_/actions/github/webhook?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
120 finished with status 3
Git package ERRORS = 2
error: Invalid argument(s). Unable to get action 'combinators/eca': Get https://nil:nil/api/v1/namespaces/_/actions/combinators/eca?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
122 finished with status 3
error: Invalid argument(s). Unable to get action 'slack/post': Get https://nil:nil/api/v1/namespaces/_/actions/slack/post?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
141 finished with status 3
Slack package ERRORS = 2
error: Invalid argument(s). Unable to get action 'samples/curl': Get https://nil:nil/api/v1/namespaces/_/actions/samples/curl?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
error: Invalid argument(s). Unable to get action 'samples/greeting': Get https://nil:nil/api/v1/namespaces/_/actions/samples/greeting?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
error: Invalid argument(s). Unable to get action 'utils/head': Get https://nil:nil/api/v1/namespaces/_/actions/utils/head?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
error: Invalid argument(s). Unable to get action 'websocket/send': Get https://nil:nil/api/v1/namespaces/_/actions/websocket/send?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
error: Invalid argument(s). Unable to get action 'utils/echo': Get https://nil:nil/api/v1/namespaces/_/actions/utils/echo?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
155 finished with status 3
121 finished with status 3
WebSocket package ERRORS = 2
error: Invalid argument(s). Unable to get action 'samples/wordCount': Get https://nil:nil/api/v1/namespaces/_/actions/samples/wordCount?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
error: Invalid argument(s). Unable to get action 'utils/smash': Get https://nil:nil/api/v1/namespaces/_/actions/utils/smash?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
error: Invalid argument(s). Unable to get action 'utils/split': Get https://nil:nil/api/v1/namespaces/_/actions/utils/split?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
error: Invalid argument(s). Unable to get action 'weather/forecast': Get https://nil:nil/api/v1/namespaces/_/actions/weather/forecast?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
error: Invalid argument(s). Unable to get action 'samples/helloWorld': Get https://nil:nil/api/v1/namespaces/_/actions/samples/helloWorld?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
154 finished with status 3
Weather package ERRORS = 2
error: Invalid argument(s). Unable to get action 'utils/cat': Get https://nil:nil/api/v1/namespaces/_/actions/utils/cat?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
error: Invalid argument(s). Unable to get action 'combinators/forwarder': Get https://nil:nil/api/v1/namespaces/_/actions/combinators/forwarder?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
156 finished with status 3
160 finished with status 3
123 finished with status 3
124 finished with status 3
125 finished with status 3
combinator package ERRORS = 5
161 finished with status 3
24 finished with status 5
25 finished with status 2
26 finished with status 2
error: Invalid argument(s). Unable to get action 'utils/sort': Get https://nil:nil/api/v1/namespaces/_/actions/utils/sort?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
166 finished with status 3
167 finished with status 3
error: Invalid argument(s). Unable to get action 'utils/namespace': Get https://nil:nil/api/v1/namespaces/_/actions/utils/namespace?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
error: Invalid argument(s). Unable to get action 'utils/date': Get https://nil:nil/api/v1/namespaces/_/actions/utils/date?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
169 finished with status 3
172 finished with status 3
error: Invalid argument(s). Unable to get action 'watson-textToSpeech/textToSpeech': Get https://nil:nil/api/v1/namespaces/_/actions/watson-textToSpeech/textToSpeech?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
error: Invalid argument(s). Unable to get action 'utils/hosturl': Get https://nil:nil/api/v1/namespaces/_/actions/utils/hosturl?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
176 finished with status 3
180 finished with status 3
181 finished with status 3
185 finished with status 3
189 finished with status 3
whisk.system entities ERRORS = 15
27 finished with status 15
error: Invalid argument(s). Unable to get action 'watson-speechToText/speechToText': Get https://nil:nil/api/v1/namespaces/_/actions/watson-speechToText/speechToText?code=false: invalid URL port "nil"
Run 'wsk --help' for usage.
209 finished with status 3
214 finished with status 3
Watson package ERRORS = 4
28 finished with status 4
29 finished with status 2
30 finished with status 2
open catalog ERRORS = 7

mycluster.yaml

jinc@jinc-MacBookPro:~/github/openwhisk-deploy-kube$ cat mycluster.yaml 
whisk:
  ingress:
    type: NodePort
    apiHostName: 10.192.0.4
    apiHostPort: 31001

nginx:
  httpsNodePort: 31001

invoker:
  containerFactory:
    dind: true
daisy-ycguo commented 5 years ago

The environment whisk_api_host_nameAndPort you passed to this job is nil:nil. That caused the failure.

Will you check the line 34 of your local helm/openwhisk/templates/clusterConfigMaps.yaml? Recently there is an update to change api_host_name to apiHostName. I see you are using the latest variable name in your mycluster.yaml. I wonder if you have updated this file clusterConfigMaps.yaml to the latest version.

whisk_api_host_nameAndPort: "{{ .Values.whisk.ingress.apiHostName }}:{{ .Values.whisk.ingress.apiHostPort }}"
JiniousChoi commented 5 years ago

Actually, after I opened this issue I did like what you suggest by myself. nil:nil kinda gave it away. I looked into recent commits on the project and realized that there was an update on the clusterConfigMaps.yaml. But, no luck. Mine was up-to-date.

I did a few things w.r.t. helm because it kinda seemed related. (the yaml files under $PROJECT/helm directory seem to be parsed, interpolated and executed by helm, aren't they?) And now it's working! But I don't know why because I did 2 things.

  1. Originally I installed helm by snap with classic switch. So I purged it and reinstalled the binary from the official website manually. (same latest version 2.11.0)
  2. Do I ever need to rm -rf $HOME/.helm on every kubeadm-dind-cluster running? I never did so, and I thought I had to give it a try. Just in case it got screwed up inside.

Any known issues with what I have done?

dgrove-oss commented 5 years ago

not sure exactly why it is working for you now and wasn't before. I'd guess you somehow had only some of the changes from api_host_name to apiHostName, but I don't know exactly how that happened.

You do not need to rm -rf $HOME/.helm each time you start the cluster (although you need to do the helm init command, which maybe is the same thing)

JiniousChoi commented 5 years ago

Thanks, I always did helm init on every cluster setup. I have no idea.