compose pods in CrashLoopBackOff with minikube

[neerolyte]

Attempting to try compose-on-kubernetes with a minikube I have followed https://github.com/docker/compose-on-kubernetes/blob/master/docs/install-on-minikube.md .

I have ended up rebuilding minikube with 8 vCPUs and 8GB of RAM to see if it was a VM resources issue, so this is from a fresh install.

Minikube already seemed to have an etcd so I skipped manually installing it:

$ kubectl get pods --all-namespaces | grep etcd
kube-system   etcd-minikube                          1/1     Running            0          17h

The installer appears to run ok:

$ ./installer-linux -namespace=compose -namespace=compose -etcd-servers=http://compose-etcd-client:2379 -tag=v0.4.18
INFO[0000] Checking installation state                  
INFO[0000] Install image with tag "v0.4.18" in namespace "compose" 
INFO[0000] Api server: image: "docker/kube-compose-api-server:v0.4.18", pullPolicy: "Always" 
INFO[0000] Controller: image: "docker/kube-compose-controller:v0.4.18", pullPolicy: "Always" 

I'm unclear from the instructions if we're supposed to replacing the -etcd-servers parameter, but it didn't error.

The next step is to check that Compose on Kubernetes is installed:

$ kubectl api-versions | grep compose
# returns nothing

At least one of the compose pods generally shows up as Error or CrashLoopBackOff:

$ kubectl get pods --namespace=compose
NAME                           READY   STATUS             RESTARTS   AGE
compose-68d845b598-95d42       1/1     Running            4          5m9s
compose-api-5f4b9d785c-lfppl   0/1     CrashLoopBackOff   4          5m9s

It looks like the liveness probe is failing too many times:

$ kubectl get events -w --namespace=compose
LAST SEEN   TYPE      REASON              KIND         MESSAGE
3m57s       Normal    Scheduled           Pod          Successfully assigned compose/compose-68d845b598-95d42 to minikube
101s        Normal    Pulling             Pod          pulling image "docker/kube-compose-controller:v0.4.18"
98s         Normal    Pulled              Pod          Successfully pulled image "docker/kube-compose-controller:v0.4.18"
98s         Normal    Created             Pod          Created container
98s         Normal    Started             Pod          Started container
52s         Warning   Unhealthy           Pod          Liveness probe failed: Get http://172.17.0.6:8080/healthz: dial tcp 172.17.0.6:8080: connect: connection refused
101s        Normal    Killing             Pod          Killing container with id docker://compose:Container failed liveness probe.. Container will be killed and recreated.
3m57s       Normal    SuccessfulCreate    ReplicaSet   Created pod: compose-68d845b598-95d42
3m57s       Normal    Scheduled           Pod          Successfully assigned compose/compose-api-5f4b9d785c-lfppl to minikube
94s         Normal    Pulling             Pod          pulling image "docker/kube-compose-api-server:v0.4.18"
90s         Normal    Pulled              Pod          Successfully pulled image "docker/kube-compose-api-server:v0.4.18"
90s         Normal    Created             Pod          Created container
90s         Normal    Started             Pod          Started container
72s         Warning   Unhealthy           Pod          Liveness probe failed: Get http://172.17.0.5:8080/healthz: dial tcp 172.17.0.5:8080: connect: connection refused
109s        Warning   BackOff             Pod          Back-off restarting failed container
3m57s       Normal    SuccessfulCreate    ReplicaSet   Created pod: compose-api-5f4b9d785c-lfppl
3m57s       Normal    ScalingReplicaSet   Deployment   Scaled up replica set compose-api-5f4b9d785c to 1
3m57s       Normal    ScalingReplicaSet   Deployment   Scaled up replica set compose-68d845b598 to 1

[neerolyte]

Ok my issue here stems from the statement at https://github.com/docker/compose-on-kubernetes/blob/5fe7240/docs/install-on-minikube.md#deploy-etcd :

If you already have an etcd instance, skip this.

There is an etcd in kube-system but apparently Compose on Kubernetes is not looking for that one and I do have to run through the How to deploy etcd instructions.

It would be good to clear up the statement about skipping etcd - as a newbie I assumed the etcd in kube-system was sufficient for me to skip adding in another one.

[ijc]

Thanks for your report. I raised some docs clarifications in #62 which I hope will help.

[neerolyte]

Yep that makes much more sense, thanks :)

[akin-ozer]

Deleting part of deployment config file "health-checkz" and following "8080" flag form both failing pods fixes this issue. I think "installer-linux" not correctly configured because "healt-checkz" flag doesn't exists. You can trace this issue from failed pods.