Closed JiniousChoi closed 5 years ago
Could you do kubectl version
(or paste in the output of minikube startup) to verify what version of Kubernetes is actually running in your minikube? One common cause of nginx going into a crash loop on startup is a Kubernetes bug with volume mount subpaths that is present in Kubernetes 1.9.4.
Of course.
~ jinchoi$ kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-04-10T12:46:31Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"linux/amd64"}
It's not clear to me whether Kubernetes 1.10.0 had the buggy volume mount subpath (https://github.com/kubernetes/kubernetes/issues/61076). It was fixed around the time of the 1.10.0 release, but the 1.10.x release notes (https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.10.md) have fixes for subpath mounting in versions up to 1.10.3. I'd suggest trying Kubernetes 1.10.5 (which is what we test in travis) and see if you still see the nginx crash loop.
@dgrove-oss you mean 1.10.5 from travis, not 1.10.3 right?
yes. please try matching one of our travisCI "known good" combinations and use Kubernetes 1.10.5 with minikube 0.28.2.
@JiniousChoi - did switching to minikube 0.28.2 / kubernetes 1.10.5 resolve the crash for you?
I had the same host not found in resolver "kube-dns.kube-system"
as kube-dns was replaced by https://coredns.io/
Changing the nginx config in nginxConfigMap.yaml to use resolver coredns.kube-system
solved this for me.
Thanks @dedene -- I had completely forgotten we had hardwired the resolver in nginxConfigMap.yaml. I will look for a fix on Monday.
I am suffering from a new issue with "switching to minikube 0.28.2 / kubernetes 1.10.5" on my Mac. I'll log the error message here soon. Also, I am planning to give it a try on my new ubuntu machine.
I tried this project on my new Ubuntu 18.04 LtS and now I have my controller node CrashLoopbackOff
My environmen and what commands I ran:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic
$ docker --version
Docker version 18.06.1-ce, build e68fc7a
$ ./dind-cluster-v1.10.sh up
$ sudo snap install helm --classic
$ helm version
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
$ helm init
$ kubectl get pods -n kube-system
$ kubectl create clusterrolebinding tiller-cluster-admin --clusterrole=cluster-admin --serviceaccount=kube-system:default
$ kubectl create namespace openwhisk
$ kubectl label nodes --all openwhisk-role=invoker
$ kubectl describe node kube-node-2 | grep InternalIP
InternalIP: 10.192.0.4
$ cat mycluster.yaml
whisk:
ingress:
type: NodePort
api_host_name: 10.192.0.4
api_host_port: 31001
nginx:
httpsNodePort: 31001
$ helm install ./helm/openwhisk --namespace=openwhisk --name=owdev -f mycluster.yaml
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
4b0464a65cc9 mirantis/kubeadm-dind-cluster:v1.10 "/sbin/dind_init sys…" About an hour ago Up About an hour 8080/tcp kube-node-2
bfa123bf0fc2 mirantis/kubeadm-dind-cluster:v1.10 "/sbin/dind_init sys…" About an hour ago Up About an hour 8080/tcp kube-node-1
b74ccbddd47a mirantis/kubeadm-dind-cluster:v1.10 "/sbin/dind_init sys…" About an hour ago Up About an hour 127.0.0.1:32768->8080/tcp kube-master
And the result is:
$ helm status owdev
LAST DEPLOYED: Tue Oct 23 09:45:58 2018
NAMESPACE: openwhisk
STATUS: DEPLOYED
RESOURCES:
==> v1/StatefulSet
NAME AGE
controller 24m
kafka 24m
zookeeper 24m
==> v1/Pod(related)
NAME READY STATUS RESTARTS AGE
invoker-5wb7f 0/1 Init:0/2 0 24m
invoker-sw5hk 0/1 Init:0/2 0 24m
apigateway-7b87dd957f-r4x4v 1/1 Running 0 24m
couchdb-6b87ccfb78-m8gnh 1/1 Running 0 24m
nginx-598c75d7d6-9xnw2 1/1 Running 0 24m
redis-5d77674f65-b4vqg 1/1 Running 0 24m
controller-0 0/1 CrashLoopBackOff 7 24m
kafka-0 1/1 Running 0 24m
zookeeper-0 1/1 Running 0 24m
init-couchdb-s9xdx 0/1 Completed 0 24m
==> v1/Deployment
NAME AGE
apigateway 24m
couchdb 24m
nginx 24m
redis 24m
==> v1/Job
init-couchdb 24m
install-catalog 24m
install-routemgmt 24m
==> v1/Secret
db.auth 24m
whisk.auth 24m
docker.registry.auth 24m
nginx 24m
==> v1/ConfigMap
db.config 24m
whisk.config 24m
init-couchdb 24m
install-catalog 24m
install-routemgmt 24m
docker-pull-runtimes 24m
nginx 24m
zookeeper 24m
==> v1/ServiceAccount
ow-core 24m
==> v1/RoleBinding
allow-view 24m
==> v1/Service
apigateway 24m
controller 24m
couchdb 24m
kafka 24m
nginx 24m
redis 24m
zookeeper 24m
==> v1/DaemonSet
invoker 24m
Somehow I missed getting this into the main README (it was buried in the troubleshooting docs). The command to start the dind cluster is:
USE_HAIRPIN=true ./dind-cluster-v1.10.sh up
If you don't set USE_HAIRPIN=true, then kafka can't talk to itself, so it doesn't come up cleanly and the controller goes into a crash loop because it can't create kafka topics.
Submitted docs fix in #320
I am trying to deploy openwhisk on minikube. I followed all the instructions in README.md in the depth-first-search manner, I mean, for the further instructions links. No luck.
All I need is 'hello world'-kind-of introduction for it. No fancy configuration is needed for me yet.
Below are my macbook environment and logs and descriptions for the components with issues. Please advise.
Nginx
Controller
Invoker
Kafka