docker / compose-on-kubernetes

Deploy applications described in Compose onto Kubernetes clusters
Apache License 2.0
1.42k stars 159 forks source link

Error: the server is currently unable to handle the request (post stacks.compose.docker.com) #58

Closed collabnix closed 3 years ago

collabnix commented 5 years ago

Pre-requisite

Verifying Minikube Status

🚩 >  minikube status
host: Running
kubelet: Running
apiserver: Running
kubectl: Correctly Configured: pointing to minikube-vm at 192.168.99.100

Verifying Current Context

kubectl config current-context
minikube

Listing out Services running in Minikube

🚩 >  minikube service list
|-------------|-----------------------|--------------|
|  NAMESPACE  |         NAME          |     URL      |
|-------------|-----------------------|--------------|
| compose     | compose-api           | No node port |
| compose     | compose-etcd          | No node port |
| compose     | compose-etcd-client   | No node port |
| compose     | etcd-restore-operator | No node port |
| default     | kubernetes            | No node port |
| kube-system | kube-dns              | No node port |
| kube-system | kubernetes-dashboard  | No node port |
| kube-system | tiller-deploy         | No node port |
|-------------|-----------------------|--------------|
kubectl api-versions| grep compose
compose.docker.com/v1beta1
compose.docker.com/v1beta2

Verifying Docker Version

🚩 >  docker version
Client: Docker Engine - Community
 Version:           18.09.1
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        4c52b90
 Built:             Wed Jan  9 19:33:12 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.1
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       4c52b90
  Built:            Wed Jan  9 19:41:49 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 Kubernetes:
  Version:          v1.12.4
  StackAPI:         v1beta2

Create docker-compose file

version: '3.4'
services:
  web1:
    image: nginx:alpine
    ports:
      - "81:80"
    deploy:
      replicas: 3
  db1:
    image: redis:alpine
    deploy:
      replicas: 2

Listing out the stack

docker stack ls
the server is currently unable to handle the request (get stacks.compose.docker.com)
 docker stack deploy --orchestrator=kubernetes -c docker-compose.yml
the server is currently unable to handle the request (post stacks.compose.docker.com)
  kubectl get all --namespace compose
NAME                                                                 READY     STATUS             RESTARTS   AGE
po/compose-api-7f95fcd458-x8zrn                                      0/1       CrashLoopBackOff   7          19m
po/compose-d4696764f-fh6tq                                           1/1       Running            8          19m
po/etcd-operator-etcd-operator-etcd-backup-operator-7978f8bc4hbrk9   1/1       Running            0          15m
po/etcd-operator-etcd-operator-etcd-operator-6c57fff9d5-772w9        1/1       Running            0          15m
po/etcd-operator-etcd-operator-etcd-restore-operator-6d787599pz788   1/1       Running            0          15m

NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
svc/compose-api             ClusterIP   10.100.245.80    <none>        443/TCP             19m
svc/compose-etcd            ClusterIP   None             <none>        2379/TCP,2380/TCP   14m
svc/compose-etcd-client     ClusterIP   10.107.111.166   <none>        2379/TCP            14m
svc/etcd-restore-operator   ClusterIP   10.108.77.217    <none>        19999/TCP           15m

NAME                                                       DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
deploy/compose                                             1         1         1            1           19m
deploy/compose-api                                         1         1         1            0           19m
deploy/etcd-operator-etcd-operator-etcd-backup-operator    1         1         1            1           15m
deploy/etcd-operator-etcd-operator-etcd-operator           1         1         1            1           15m
deploy/etcd-operator-etcd-operator-etcd-restore-operator   1         1         1            1           15m

NAME                                                              DESIRED   CURRENT   READY     AGE
rs/compose-api-7f95fcd458                                         1         1         0         19m
rs/compose-d4696764f                                              1         1         1         19m
rs/etcd-operator-etcd-operator-etcd-backup-operator-7978f8bc47    1         1         1         15m
rs/etcd-operator-etcd-operator-etcd-operator-6c57fff9d5           1         1         1         15m
rs/etcd-operator-etcd-operator-etcd-restore-operator-6d787599f8   1         1         1         15m
[Captains-Bay]🚩 >
ubectl logs --namespace compose po/compose-api-7f95fcd458-x8zrn
ERROR: logging before flag.Parse: I0118 15:58:53.147046       1 plugins.go:158] Loaded 2 mutating admission controller(s) successfully in the following order: NamespaceLifecycle,MutatingAdmissionWebhook.
ERROR: logging before flag.Parse: I0118 15:58:53.148052       1 plugins.go:161] Loaded 1 validating admission controller(s) successfully in the following order: ValidatingAdmissionWebhook.
ERROR: logging before flag.Parse: W0118 15:58:53.148642       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ERROR: logging before flag.Parse: F0118 15:59:13.173387       1 storage_decorator.go:57] Unable to create storage backend: config (&{ /registry/docker.com/stacks [http://compose-etcd-client:2379]    true false 0 0xc000461900 <nil> 5m0s 1m0s}), err (dial tcp 10.107.111.166:2379: i/o timeout)

Any idea? It was working for me few minutes back.

simonferquel commented 5 years ago

Something really strange has happened: the services exposing your etcd are there, but there is no etcd pod in your environment except for etcd operator infrastructure ones)

wsilva commented 5 years ago

I faced a similar issue a couple times. I figured out that I was trying to deploy a new etcd cluster before etcd operator is up and running.

Both times I fixed running manually the following commands to create the etcd cluster used by compose:

$ cat > compose-etcd.yaml << EOF
apiVersion: "etcd.database.coreos.com/v1beta2"
kind: "EtcdCluster"
metadata:
  name: "compose-etcd"
  namespace: "compose"
spec:
  size: 3
  version: "3.2.13"
EOF
$ kubectl apply -f compose-etcd.yaml

Then, after the pods running I run the following to reinstall it:

$ ./installer-darwin -namespace=compose -uninstall
$ ./installer-darwin -namespace=compose -etcd-servers=http://compose-etcd-client:2379 -tag=v0.4.18

And then everything worked fine.

Not sure if it is the same issue but it worth a try.

tienvx commented 5 years ago

I faced the same problem. This is how I fix it:

  1. Delete the minikube cluster by running minikube delete
  2. Create new compose-etcd.yaml file using the new content in the doc
  3. Follow step by step in https://collabnix.com/a-first-look-at-compose-on-kubernetes-for-minikube/, but this time use the new compose-etcd.yaml file above. This is the last step:
    docker stack deploy --orchestrator=kubernetes -c docker-compose.yml mbt
  4. I can list stacks, and remove them
    docker stack ls --orchestrator=kubernetes
    docker stack rm mbt --orchestrator=kubernetes

Update: After remove stack and restart minikube cluster by running:

docker stack rm mbt --orchestrator=kubernetes
minikube stop
minikube start

the problem is still persist. I can not deploy the stack again. Here are error messages:

chris-crone commented 3 years ago

Hi! Thanks for your feedback. Unfortunately we're no longer maintaining this repository so I will close this issue.