search

apache / openwhisk-deploy-kube

The Apache OpenWhisk Kubernetes Deployment repository supports deploying the Apache OpenWhisk system on Kubernetes and OpenShift clusters.
https://openwhisk.apache.org/
Apache License 2.0
297 stars 231 forks source link

Nginx CrashLoopbackOff #303

Closed JiniousChoi closed 5 years ago

JiniousChoi commented 5 years ago

I am trying to deploy openwhisk on minikube. I followed all the instructions in README.md in the depth-first-search manner, I mean, for the further instructions links. No luck.

All I need is 'hello world'-kind-of introduction for it. No fancy configuration is needed for me yet.

Below are my macbook environment and logs and descriptions for the components with issues. Please advise.

~ jinchoi$ system_profiler SPSoftwareDataType
Software:

    System Software Overview:

      System Version: macOS 10.13.3 (17D47)
      Kernel Version: Darwin 17.4.0
      Boot Volume: Macintosh HD
      Boot Mode: Normal
      Computer Name: Jin's MacBook Pro
      User Name: jinchoi (jinchoi)
      Secure Virtual Memory: Enabled
      System Integrity Protection: Enabled
      Time since boot: 20:59

~ jinchoi$ asdf list
kubectl
  1.10.5
  1.9.0
minikube
  0.25.2
  0.28.2
~ jinchoi$ asdf current minikube
0.25.2  (set by /Users/jinchoi/.tool-versions)
~ jinchoi$ asdf current kubectl
1.9.0   (set by /Users/jinchoi/.tool-versions)

~ jinchoi$ helm version
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
~ jinchoi$ ifconfig | grep inet
    inet 127.0.0.1 netmask 0xff000000 
        ... omit ...
    inet 192.168.99.1 netmask 0xffffff00 broadcast 192.168.99.255
incubator-openwhisk-deploy-kube jinchoi$ tree -L 1
.
├── CONTRIBUTING.md
├── DISCLAIMER.txt
├── LICENSE.txt
├── NOTICE.txt
├── README.md
├── docker
├── docs
├── helm
├── mycluster.yaml
└── tools
4 directories, 6 files
incubator-openwhisk-deploy-kube jinchoi$ cat mycluster.yaml 
whisk:
  ingress:
    type: NodePort
    api_host_name: 192.168.99.100
    api_host_port: 31001

nginx:
  httpsNodePort: 31001
~ jinchoi$ kubectl get pods --namespace=openwhisk
NAME                          READY     STATUS             RESTARTS   AGE
apigateway-7b87dd957f-5dngm   1/1       Running            0          19h
controller-0                  0/1       Init:0/2           0          19h
couchdb-6b87ccfb78-tgtsx      1/1       Running            0          19h
invoker-p8hdn                 0/1       Init:1/2           0          19h
kafka-0                       0/1       Init:0/1           0          19h
nginx-598c75d7d6-ztpvs        0/1       CrashLoopBackOff   213        19h
redis-5d77674f65-5d2jb        1/1       Running            0          19h
zookeeper-0

Nginx

~ jinchoi$ kubectl -n openwhisk logs nginx-598c75d7d6-ztpvs
2018/09/27 23:33:48 [emerg] 1#1: host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:41
nginx: [emerg] host not found in resolver "kube-dns.kube-system" in /etc/nginx/nginx.conf:41
~ jinchoi$ kubectl -n openwhisk describe pod nginx-598c75d7d6-ztpvs
Name:           nginx-598c75d7d6-ztpvs
Namespace:      openwhisk
Node:           minikube/192.168.99.100
Start Time:     Thu, 27 Sep 2018 13:26:26 +0900
Labels:         name=nginx
                pod-template-hash=1547318382
Annotations:    <none>
Status:         Running
IP:             172.17.0.9
Controlled By:  ReplicaSet/nginx-598c75d7d6
Containers:
  nginx:
    Container ID:   docker://644b471d740d1dfa518ce8bee7bcc4bdf09913b269ab551e13113a092fbb74fa
    Image:          nginx:1.11
    Image ID:       docker-pullable://nginx@sha256:e6693c20186f837fc393390135d8a598a96a833917917789d63766cab6c59582
    Ports:          80/TCP, 443/TCP, 8443/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Fri, 28 Sep 2018 08:39:12 +0900
      Finished:     Fri, 28 Sep 2018 08:39:32 +0900
    Ready:          False
    Restart Count:  215
    Environment:    <none>
    Mounts:
      /etc/nginx/certs from nginx-certs (rw)
      /etc/nginx/nginx.conf from nginx-conf (rw)
      /logs from logs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wlxzb (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  nginx-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  nginx
    Optional:    false
  nginx-conf:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      nginx
    Optional:  false
  logs:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  default-token-wlxzb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wlxzb
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason   Age                  From               Message
  ----     ------   ----                 ----               -------
  Normal   Pulled   34m (x209 over 19h)  kubelet, minikube  Container image "nginx:1.11" already present on machine
  Warning  BackOff  4m (x4832 over 19h)  kubelet, minikube  Back-off restarting failed container

Controller

~ jinchoi$ kubectl -n openwhisk logs controller-0
Error from server (BadRequest): container "controller" in pod "controller-0" is waiting to start: PodInitializing
~ jinchoi$ kubectl -n openwhisk describe pod controller-0
Name:           controller-0
Namespace:      openwhisk
Node:           minikube/192.168.99.100
Start Time:     Thu, 27 Sep 2018 13:26:26 +0900
Labels:         controller-revision-hash=controller-5bc6758f68
                name=controller
                statefulset.kubernetes.io/pod-name=controller-0
Annotations:    <none>
Status:         Pending
IP:             172.17.0.5
Controlled By:  StatefulSet/controller
Init Containers:
  wait-for-kafka:
    Container ID:  docker://012b8b856f684289007a03a125bd6aa8bf1f633a68fde037a4ee2c0af3dcfdad
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd
    Port:          <none>
    Command:
      sh
      -c
      result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 zookeeper-0.zookeeper.openwhisk.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi done; echo "Zookeeper is up; will wait for 10 seconds to give kafka time to initialize"; sleep 10;
    State:          Running
      Started:      Thu, 27 Sep 2018 13:28:09 +0900
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-5pfz4 (ro)
  wait-for-couchdb:
    Container ID:  
    Image:         busybox
    Image ID:      
    Port:          <none>
    Command:
      sh
      -c
      while true; do echo 'checking CouchDB readiness'; wget -T 5 --spider $READINESS_URL --header="Authorization: Basic d2hpc2tfYWRtaW46c29tZV9wYXNzdzByZA=="; result=$?; if [ $result -eq 0 ]; then echo 'Success: CouchDB is ready!'; break; fi; echo '...not ready yet; sleeping 3 seconds before retry'; sleep 3; done;
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      READINESS_URL:  http://couchdb.openwhisk.svc.cluster.local:5984/ow_kube_couchdb_initialized_marker
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-5pfz4 (ro)
Containers:
  controller:
    Container ID:  
    Image:         openwhisk/controller:latest
    Image ID:      
    Ports:         8080/TCP, 2552/TCP, 19999/TCP
    Command:
      /bin/bash
      -c
      /init.sh `hostname | cut -d'-' -f2`
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:8080/ping delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      PORT:                       8080
      CONFIG_whisk_info_date:     <set to the key 'whisk_info_date' of config map 'whisk.config'>     Optional: false
      CONFIG_whisk_info_buildNo:  <set to the key 'whisk_info_buildNo' of config map 'whisk.config'>  Optional: false
      JAVA_OPTS:                  -Xmx1024M 
      CONTROLLER_OPTS:            
      RUNTIMES_MANIFEST:          {
    "runtimes": {
        "nodejs": [
            {
                "kind": "nodejs",
                "image": {
                    "prefix": "openwhisk",
                    "name": "nodejsaction",
                    "tag": "latest"
                },
                "deprecated": true
            },
            {
                "kind": "nodejs:6",
                "default": true,
                "image": {
                    "prefix": "openwhisk",
                    "name": "nodejs6action",
                    "tag": "latest"
                },
                "deprecated": false,
                "stemCells": [{
                    "count": 2,
                    "memory": "256 MB"
                }]
            },
            {
                "kind": "nodejs:8",
                "default": false,
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-nodejs-v8",
                    "tag": "latest"
                },
                "deprecated": false
            }
        ],
        "python": [
            {
                "kind": "python",
                "image": {
                    "prefix": "openwhisk",
                    "name": "python2action",
                    "tag": "latest"
                },
                "deprecated": false
            },
            {
                "kind": "python:2",
                "default": true,
                "image": {
                    "prefix": "openwhisk",
                    "name": "python2action",
                    "tag": "latest"
                },
                "deprecated": false
            },
            {
                "kind": "python:3",
                "image": {
                    "prefix": "openwhisk",
                    "name": "python3action",
                    "tag": "latest"
                },
                "deprecated": false
            }
        ],
        "swift": [
            {
                "kind": "swift:3.1.1",
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-swift-v3.1.1",
                    "tag": "latest"
                },
                "deprecated": false
            },
            {
                "kind": "swift:4.1",
                "default": true,
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-swift-v4.1",
                    "tag": "latest"
                },
                "deprecated": false
            }
        ],
        "java": [
            {
                "kind": "java",
                "default": true,
                "image": {
                    "prefix": "openwhisk",
                    "name": "java8action",
                    "tag": "latest"
                },
                "deprecated": false,
                "attached": {
                    "attachmentName": "jarfile",
                    "attachmentType": "application/java-archive"
                },
                "sentinelledLogs": false,
                "requireMain": true
            }
        ],
        "php": [
            {
                "kind": "php:7.1",
                "default": true,
                "deprecated": false,
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-php-v7.1",
                    "tag": "latest"
                }
            }
        ],
        "ruby": [
            {
                "kind": "ruby:2.5",
                "default": true,
                "deprecated": false,
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-ruby-v2.5",
                    "tag": "latest"
                }
            }
        ]
    },
    "blackboxes": [
        {
            "prefix": "openwhisk",
            "name": "dockerskeleton",
            "tag": "latest"
        }
    ]
}

      CONFIG_whisk_loadbalancer_invokerUserMemory:     2048m
      KAFKA_HOSTS:                                     kafka.openwhisk.svc.cluster.local:9092
      KAFKA_HOST_PORT:                                 9092
      CONFIG_whisk_couchdb_username:                   <set to the key 'db_username' in secret 'db.auth'>        Optional: false
      CONFIG_whisk_couchdb_password:                   <set to the key 'db_password' in secret 'db.auth'>        Optional: false
      CONFIG_whisk_couchdb_port:                       <set to the key 'db_port' of config map 'db.config'>      Optional: false
      CONFIG_whisk_couchdb_protocol:                   <set to the key 'db_protocol' of config map 'db.config'>  Optional: false
      CONFIG_whisk_couchdb_host:                       couchdb.openwhisk.svc.cluster.local
      CONFIG_whisk_couchdb_provider:                   <set to the key 'db_provider' of config map 'db.config'>           Optional: false
      CONFIG_whisk_couchdb_databases_WhiskActivation:  <set to the key 'db_whisk_activations' of config map 'db.config'>  Optional: false
      CONFIG_whisk_couchdb_databases_WhiskEntity:      <set to the key 'db_whisk_actions' of config map 'db.config'>      Optional: false
      CONFIG_whisk_couchdb_databases_WhiskAuth:        <set to the key 'db_whisk_auths' of config map 'db.config'>        Optional: false
      LIMITS_ACTIONS_SEQUENCE_MAXLENGTH:               50
      LIMITS_TRIGGERS_FIRES_PERMINUTE:                 60
      LIMITS_ACTIONS_INVOKES_PERMINUTE:                60
      LIMITS_ACTIONS_INVOKES_CONCURRENT:               30
      CONTROLLER_INSTANCES:                            1
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from ow-core-token-5pfz4 (ro)
Conditions:
  Type           Status
  Initialized    False 
  Ready          False 
  PodScheduled   True 
Volumes:
  ow-core-token-5pfz4:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  ow-core-token-5pfz4
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

Invoker

~ jinchoi$ kubectl -n openwhisk logs invoker-p8hdn
Error from server (BadRequest): container "invoker" in pod "invoker-p8hdn" is waiting to start: PodInitializing
~ jinchoi$ kubectl -n openwhisk describe pod invoker-p8hdn
Name:           invoker-p8hdn
Namespace:      openwhisk
Node:           minikube/192.168.99.100
Start Time:     Thu, 27 Sep 2018 13:26:26 +0900
Labels:         controller-revision-hash=3561614914
                name=invoker
                pod-template-generation=1
Annotations:    <none>
Status:         Pending
IP:             172.17.0.3
Controlled By:  DaemonSet/invoker
Init Containers:
  docker-pull-runtimes:
    Container ID:   docker://5641ffdd538bfbd797f6487f5086e8d474a3e15dab020edd1e783716ab4656a3
    Image:          openwhisk/ansible-runner:latest
    Image ID:       docker-pullable://openwhisk/ansible-runner@sha256:825fdd7d536d92871fb5b2c2a1ae5a1eb1baae359aabbf4ebf94f0e244044b80
    Port:           <none>
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 27 Sep 2018 13:27:01 +0900
      Finished:     Thu, 27 Sep 2018 13:35:00 +0900
    Ready:          True
    Restart Count:  0
    Environment:
      RUNTIMES_MANIFEST:  {
    "runtimes": {
        "nodejs": [
            {
                "kind": "nodejs",
                "image": {
                    "prefix": "openwhisk",
                    "name": "nodejsaction",
                    "tag": "latest"
                },
                "deprecated": true
            },
            {
                "kind": "nodejs:6",
                "default": true,
                "image": {
                    "prefix": "openwhisk",
                    "name": "nodejs6action",
                    "tag": "latest"
                },
                "deprecated": false,
                "stemCells": [{
                    "count": 2,
                    "memory": "256 MB"
                }]
            },
            {
                "kind": "nodejs:8",
                "default": false,
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-nodejs-v8",
                    "tag": "latest"
                },
                "deprecated": false
            }
        ],
        "python": [
            {
                "kind": "python",
                "image": {
                    "prefix": "openwhisk",
                    "name": "python2action",
                    "tag": "latest"
                },
                "deprecated": false
            },
            {
                "kind": "python:2",
                "default": true,
                "image": {
                    "prefix": "openwhisk",
                    "name": "python2action",
                    "tag": "latest"
                },
                "deprecated": false
            },
            {
                "kind": "python:3",
                "image": {
                    "prefix": "openwhisk",
                    "name": "python3action",
                    "tag": "latest"
                },
                "deprecated": false
            }
        ],
        "swift": [
            {
                "kind": "swift:3.1.1",
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-swift-v3.1.1",
                    "tag": "latest"
                },
                "deprecated": false
            },
            {
                "kind": "swift:4.1",
                "default": true,
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-swift-v4.1",
                    "tag": "latest"
                },
                "deprecated": false
            }
        ],
        "java": [
            {
                "kind": "java",
                "default": true,
                "image": {
                    "prefix": "openwhisk",
                    "name": "java8action",
                    "tag": "latest"
                },
                "deprecated": false,
                "attached": {
                    "attachmentName": "jarfile",
                    "attachmentType": "application/java-archive"
                },
                "sentinelledLogs": false,
                "requireMain": true
            }
        ],
        "php": [
            {
                "kind": "php:7.1",
                "default": true,
                "deprecated": false,
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-php-v7.1",
                    "tag": "latest"
                }
            }
        ],
        "ruby": [
            {
                "kind": "ruby:2.5",
                "default": true,
                "deprecated": false,
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-ruby-v2.5",
                    "tag": "latest"
                }
            }
        ]
    },
    "blackboxes": [
        {
            "prefix": "openwhisk",
            "name": "dockerskeleton",
            "tag": "latest"
        }
    ]
}

    Mounts:
      /task/playbook.yml from task-dir (rw)
      /var/run/docker.sock from dockersock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wlxzb (ro)
  wait-for-controller:
    Container ID:  docker://848c8eb818082083c8d56e5234372d2e6246c8fc4dd2a919440cc337c48384e2
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd
    Port:          <none>
    Command:
      sh
      -c
      result=1; until [ $result -eq 0 ]; do echo 'Checking controller readiness'; wget -T 5 --spider $READINESS_URL; result=$?; sleep 1; done; echo 'Success: controller is ready'
    State:          Running
      Started:      Thu, 27 Sep 2018 13:35:00 +0900
    Ready:          False
    Restart Count:  0
    Environment:
      READINESS_URL:  http://controller.openwhisk.svc.cluster.local:8080/ping
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wlxzb (ro)
Containers:
  invoker:
    Container ID:  
    Image:         openwhisk/invoker:latest
    Image ID:      
    Port:          8080/TCP
    Command:
      /bin/bash
      -c
      /init.sh --uniqueName $INVOKER_NAME
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      PORT:                                          8080
      WHISK_API_HOST_PROTO:                          <set to the key 'whisk_api_host_proto' of config map 'whisk.config'>  Optional: false
      WHISK_API_HOST_PORT:                           <set to the key 'whisk_api_host_port' of config map 'whisk.config'>   Optional: false
      WHISK_API_HOST_NAME:                           <set to the key 'whisk_api_host_name' of config map 'whisk.config'>   Optional: false
      INVOKER_CONTAINER_NETWORK:                     bridge
      CONFIG_whisk_docker_containerFactory_useRunc:  false
      DOCKER_IMAGE_PREFIX:                           openwhisk
      DOCKER_IMAGE_TAG:                              latest
      DOCKER_REGISTRY:                               
      CONFIG_whisk_loadbalancer_invokerUserMemory:   2048m
      INVOKER_NAME:                                   (v1:spec.nodeName)
      JAVA_OPTS:                                     -Xmx512M 
      INVOKER_OPTS:                                    -Dwhisk.spi.ContainerFactoryProvider=whisk.core.containerpool.docker.DockerContainerFactoryProvider 
      RUNTIMES_MANIFEST:                             {
    "runtimes": {
        "nodejs": [
            {
                "kind": "nodejs",
                "image": {
                    "prefix": "openwhisk",
                    "name": "nodejsaction",
                    "tag": "latest"
                },
                "deprecated": true
            },
            {
                "kind": "nodejs:6",
                "default": true,
                "image": {
                    "prefix": "openwhisk",
                    "name": "nodejs6action",
                    "tag": "latest"
                },
                "deprecated": false,
                "stemCells": [{
                    "count": 2,
                    "memory": "256 MB"
                }]
            },
            {
                "kind": "nodejs:8",
                "default": false,
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-nodejs-v8",
                    "tag": "latest"
                },
                "deprecated": false
            }
        ],
        "python": [
            {
                "kind": "python",
                "image": {
                    "prefix": "openwhisk",
                    "name": "python2action",
                    "tag": "latest"
                },
                "deprecated": false
            },
            {
                "kind": "python:2",
                "default": true,
                "image": {
                    "prefix": "openwhisk",
                    "name": "python2action",
                    "tag": "latest"
                },
                "deprecated": false
            },
            {
                "kind": "python:3",
                "image": {
                    "prefix": "openwhisk",
                    "name": "python3action",
                    "tag": "latest"
                },
                "deprecated": false
            }
        ],
        "swift": [
            {
                "kind": "swift:3.1.1",
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-swift-v3.1.1",
                    "tag": "latest"
                },
                "deprecated": false
            },
            {
                "kind": "swift:4.1",
                "default": true,
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-swift-v4.1",
                    "tag": "latest"
                },
                "deprecated": false
            }
        ],
        "java": [
            {
                "kind": "java",
                "default": true,
                "image": {
                    "prefix": "openwhisk",
                    "name": "java8action",
                    "tag": "latest"
                },
                "deprecated": false,
                "attached": {
                    "attachmentName": "jarfile",
                    "attachmentType": "application/java-archive"
                },
                "sentinelledLogs": false,
                "requireMain": true
            }
        ],
        "php": [
            {
                "kind": "php:7.1",
                "default": true,
                "deprecated": false,
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-php-v7.1",
                    "tag": "latest"
                }
            }
        ],
        "ruby": [
            {
                "kind": "ruby:2.5",
                "default": true,
                "deprecated": false,
                "image": {
                    "prefix": "openwhisk",
                    "name": "action-ruby-v2.5",
                    "tag": "latest"
                }
            }
        ]
    },
    "blackboxes": [
        {
            "prefix": "openwhisk",
            "name": "dockerskeleton",
            "tag": "latest"
        }
    ]
}

      WHISK_LOGS_DIR:                                  
      CONFIG_whisk_info_date:                          <set to the key 'whisk_info_date' of config map 'whisk.config'>  Optional: false
      CONFIG_whisk_couchdb_username:                   <set to the key 'db_username' in secret 'db.auth'>               Optional: false
      CONFIG_whisk_couchdb_password:                   <set to the key 'db_password' in secret 'db.auth'>               Optional: false
      CONFIG_whisk_couchdb_port:                       <set to the key 'db_port' of config map 'db.config'>             Optional: false
      CONFIG_whisk_couchdb_protocol:                   <set to the key 'db_protocol' of config map 'db.config'>         Optional: false
      CONFIG_whisk_couchdb_host:                       couchdb.openwhisk.svc.cluster.local
      CONFIG_whisk_couchdb_provider:                   <set to the key 'db_provider' of config map 'db.config'>           Optional: false
      CONFIG_whisk_couchdb_databases_WhiskActivation:  <set to the key 'db_whisk_activations' of config map 'db.config'>  Optional: false
      CONFIG_whisk_couchdb_databases_WhiskEntity:      <set to the key 'db_whisk_actions' of config map 'db.config'>      Optional: false
      CONFIG_whisk_couchdb_databases_WhiskAuth:        <set to the key 'db_whisk_auths' of config map 'db.config'>        Optional: false
      KAFKA_HOSTS:                                     kafka.openwhisk.svc.cluster.local:9092
      ZOOKEEPER_HOSTS:                                 zookeeper-0.zookeeper.openwhisk.svc.cluster.local:2181
    Mounts:
      /containers from dockerrootdir (rw)
      /run/runc from runc (rw)
      /sys/fs/cgroup from cgroup (rw)
      /var/run/docker.sock from dockersock (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wlxzb (ro)
Conditions:
  Type           Status
  Initialized    False 
  Ready          False 
  PodScheduled   True 
Volumes:
  cgroup:
    Type:          HostPath (bare host directory volume)
    Path:          /sys/fs/cgroup
    HostPathType:  
  runc:
    Type:          HostPath (bare host directory volume)
    Path:          /run/runc
    HostPathType:  
  dockerrootdir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/docker/containers
    HostPathType:  
  dockersock:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/docker.sock
    HostPathType:  
  task-dir:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      docker-pull-runtimes
    Optional:  false
  default-token-wlxzb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wlxzb
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/disk-pressure:NoSchedule
                 node.kubernetes.io/memory-pressure:NoSchedule
                 node.kubernetes.io/not-ready:NoExecute
                 node.kubernetes.io/unreachable:NoExecute
Events:          <none>

Kafka

~ jinchoi$ kubectl -n openwhisk logs kafka-0
Error from server (BadRequest): container "kafka" in pod "kafka-0" is waiting to start: PodInitializing
~ jinchoi$ kubectl -n openwhisk describe pod kafka-0
Name:           kafka-0
Namespace:      openwhisk
Node:           minikube/192.168.99.100
Start Time:     Thu, 27 Sep 2018 13:26:26 +0900
Labels:         controller-revision-hash=kafka-d4cb4fbf8
                name=kafka
                statefulset.kubernetes.io/pod-name=kafka-0
Annotations:    <none>
Status:         Pending
IP:             172.17.0.13
Controlled By:  StatefulSet/kafka
Init Containers:
  wait-for-zookeeper:
    Container ID:  docker://db1c5c6cc2b6931a5925cf56f8930f87f14fe4856e9dad2e9bfdd6fbc153cbfd
    Image:         busybox
    Image ID:      docker-pullable://busybox@sha256:cb63aa0641a885f54de20f61d152187419e8f6b159ed11a251a09d115fdff9bd
    Port:          <none>
    Command:
      sh
      -c
      result=1; until [ $result -eq 0 ]; do OK=$(echo ruok | nc -w 1 zookeeper-0.zookeeper.openwhisk.svc.cluster.local 2181); if [ "$OK" == "imok" ]; then result=0; echo "zookeeper returned imok!"; else echo waiting for zookeeper to be ready; sleep 1; fi; done; echo "Success: zookeeper is up"
    State:          Running
      Started:      Thu, 27 Sep 2018 13:31:46 +0900
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wlxzb (ro)
Containers:
  kafka:
    Container ID:   
    Image:          wurstmeister/kafka:0.11.0.1
    Image ID:       
    Port:           9092/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      KAFKA_BROKER_ID:             0
      KAFKA_ADVERTISED_HOST_NAME:  kafka
      KAFKA_ADVERTISED_PORT:       9092
      KAFKA_HOST_NAME:              (v1:status.podIP)
      KAFKA_PORT:                  9092
      KAFKA_ZOOKEEPER_CONNECT:     zookeeper-0.zookeeper.openwhisk.svc.cluster.local:2181
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wlxzb (ro)
Conditions:
  Type           Status
  Initialized    False 
  Ready          False 
  PodScheduled   True 
Volumes:
  default-token-wlxzb:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wlxzb
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>
dgrove-oss commented 5 years ago

Could you do kubectl version (or paste in the output of minikube startup) to verify what version of Kubernetes is actually running in your minikube? One common cause of nginx going into a crash loop on startup is a Kubernetes bug with volume mount subpaths that is present in Kubernetes 1.9.4.

JiniousChoi commented 5 years ago

Of course.

~ jinchoi$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-04-10T12:46:31Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"linux/amd64"}
dgrove-oss commented 5 years ago

It's not clear to me whether Kubernetes 1.10.0 had the buggy volume mount subpath (https://github.com/kubernetes/kubernetes/issues/61076). It was fixed around the time of the 1.10.0 release, but the 1.10.x release notes (https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.10.md) have fixes for subpath mounting in versions up to 1.10.3. I'd suggest trying Kubernetes 1.10.5 (which is what we test in travis) and see if you still see the nginx crash loop.

JiniousChoi commented 5 years ago

@dgrove-oss you mean 1.10.5 from travis, not 1.10.3 right?

dgrove-oss commented 5 years ago

yes. please try matching one of our travisCI "known good" combinations and use Kubernetes 1.10.5 with minikube 0.28.2.

dgrove-oss commented 5 years ago

@JiniousChoi - did switching to minikube 0.28.2 / kubernetes 1.10.5 resolve the crash for you?

dedene commented 5 years ago

I had the same host not found in resolver "kube-dns.kube-system" as kube-dns was replaced by https://coredns.io/

Changing the nginx config in nginxConfigMap.yaml to use resolver coredns.kube-system solved this for me.

dgrove-oss commented 5 years ago

Thanks @dedene -- I had completely forgotten we had hardwired the resolver in nginxConfigMap.yaml. I will look for a fix on Monday.

JiniousChoi commented 5 years ago

I am suffering from a new issue with "switching to minikube 0.28.2 / kubernetes 1.10.5" on my Mac. I'll log the error message here soon. Also, I am planning to give it a try on my new ubuntu machine.

JiniousChoi commented 5 years ago

I tried this project on my new Ubuntu 18.04 LtS and now I have my controller node CrashLoopbackOff

My environmen and what commands I ran:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.1 LTS
Release:    18.04
Codename:   bionic

$ docker --version
Docker version 18.06.1-ce, build e68fc7a

$ ./dind-cluster-v1.10.sh up

$ sudo snap install helm --classic
$ helm version
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

$ helm init
$ kubectl get pods -n kube-system
$ kubectl create clusterrolebinding tiller-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
$ kubectl create namespace openwhisk
$ kubectl label nodes --all openwhisk-role=invoker
$ kubectl describe node kube-node-2 | grep InternalIP
  InternalIP:  10.192.0.4

$ cat mycluster.yaml 
whisk:
  ingress:
    type: NodePort
    api_host_name: 10.192.0.4
    api_host_port: 31001

nginx:
  httpsNodePort: 31001

$ helm install ./helm/openwhisk --namespace=openwhisk --name=owdev -f mycluster.yaml

$ docker ps
CONTAINER ID        IMAGE                                 COMMAND                  CREATED             STATUS              PORTS                       NAMES
4b0464a65cc9        mirantis/kubeadm-dind-cluster:v1.10   "/sbin/dind_init sys…"   About an hour ago   Up About an hour    8080/tcp                    kube-node-2
bfa123bf0fc2        mirantis/kubeadm-dind-cluster:v1.10   "/sbin/dind_init sys…"   About an hour ago   Up About an hour    8080/tcp                    kube-node-1
b74ccbddd47a        mirantis/kubeadm-dind-cluster:v1.10   "/sbin/dind_init sys…"   About an hour ago   Up About an hour    127.0.0.1:32768->8080/tcp   kube-master

And the result is:

$ helm status owdev 
LAST DEPLOYED: Tue Oct 23 09:45:58 2018
NAMESPACE: openwhisk
STATUS: DEPLOYED

RESOURCES:
==> v1/StatefulSet
NAME        AGE
controller  24m
kafka       24m
zookeeper   24m

==> v1/Pod(related)

NAME                         READY  STATUS            RESTARTS  AGE
invoker-5wb7f                0/1    Init:0/2          0         24m
invoker-sw5hk                0/1    Init:0/2          0         24m
apigateway-7b87dd957f-r4x4v  1/1    Running           0         24m
couchdb-6b87ccfb78-m8gnh     1/1    Running           0         24m
nginx-598c75d7d6-9xnw2       1/1    Running           0         24m
redis-5d77674f65-b4vqg       1/1    Running           0         24m
controller-0                 0/1    CrashLoopBackOff  7         24m
kafka-0                      1/1    Running           0         24m
zookeeper-0                  1/1    Running           0         24m
init-couchdb-s9xdx           0/1    Completed         0         24m

==> v1/Deployment

NAME        AGE
apigateway  24m
couchdb     24m
nginx       24m
redis       24m

==> v1/Job
init-couchdb       24m
install-catalog    24m
install-routemgmt  24m

==> v1/Secret
db.auth               24m
whisk.auth            24m
docker.registry.auth  24m
nginx                 24m

==> v1/ConfigMap
db.config             24m
whisk.config          24m
init-couchdb          24m
install-catalog       24m
install-routemgmt     24m
docker-pull-runtimes  24m
nginx                 24m
zookeeper             24m

==> v1/ServiceAccount
ow-core  24m

==> v1/RoleBinding
allow-view  24m

==> v1/Service
apigateway  24m
controller  24m
couchdb     24m
kafka       24m
nginx       24m
redis       24m
zookeeper   24m

==> v1/DaemonSet
invoker  24m
dgrove-oss commented 5 years ago

Somehow I missed getting this into the main README (it was buried in the troubleshooting docs). The command to start the dind cluster is: USE_HAIRPIN=true ./dind-cluster-v1.10.sh up If you don't set USE_HAIRPIN=true, then kafka can't talk to itself, so it doesn't come up cleanly and the controller goes into a crash loop because it can't create kafka topics.

Submitted docs fix in #320