search

vitessio / vitess

Vitess is a database clustering system for horizontal scaling of MySQL.
http://vitess.io
Apache License 2.0
18.43k stars 2.09k forks source link

vitess helm install fails: vtgate-zone1... pods are failing with CrashLoopBackOff #4372

Closed makandas closed 5 years ago

makandas commented 5 years ago

Overview of the Issue

Git clone the latest vitess code and tried :

root@tofu1:~/new/vitess/helm/vitess# helm install . --name=vitess
NAME:   vitess
LAST DEPLOYED: Sat Nov 17 04:18:53 2018
NAMESPACE: default
STATUS: DEPLOYED
........................
root@tofu1:~/new/vitess/helm/vitess# helm list
NAME            REVISION        UPDATED                         STATUS          CHART                   NAMESPACE
etc-release     1               Wed Nov 14 02:41:05 2018        DEPLOYED        etcd-operator-0.8.0     default
mariadb-release 1               Thu Nov 15 21:14:08 2018        DEPLOYED        mariadb-5.2.3           default
nodemongo1      1               Sun Sep  9 06:14:43 2018        DEPLOYED        meanchart-0.1.0         default
vitess          1               Sat Nov 17 04:18:53 2018        DEPLOYED        vitess-0.9.0            default

Reproduction Steps

Steps to reproduce this issue, example:

  1. git clone latest code: root@tofu1:~/new# git clone https://github.com/vitessio/vitess.git Cloning into 'vitess'...

  2. cd vitess/helm/vitess. 3.Modified values.yaml to mariadb:

    vttablet:
vitessTag: "latest"

flavor: "mariadb103"

mysqlImage: "mariadb:10.3.4"
...
mysqlProtocol:
    enabled: false
    username: root
    passwordSecret: password
...
  3. root@tofu1:~/new/vitess/helm/vitess# helm install . --name=vitess NAME: vitess LAST DEPLOYED: Sat Nov 17 04:18:53 2018 NAMESPACE: default STATUS: DEPLOYED ........................ root@tofu1:~/new/vitess/helm/vitess# helm list NAME REVISION UPDATED STATUS CHART NAMESPACE etc-release 1 Wed Nov 14 02:41:05 2018 DEPLOYED etcd-operator-0.8.0 default mariadb-release 1 Thu Nov 15 21:14:08 2018 DEPLOYED mariadb-5.2.3 default nodemongo1 1 Sun Sep 9 06:14:43 2018 DEPLOYED meanchart-0.1.0 default vitess 1 Sat Nov 17 04:18:53 2018 DEPLOYED vitess-0.9.0 default

root@tofu1:~/new/vitess/helm/vitess# kubectl get pods NAME READY STATUS RESTARTS AGE etc-release-etcd-operator-etcd-backup-operator-7f85b5cf77-n5hrw 1/1 Running 0 3d etc-release-etcd-operator-etcd-operator-6d484f6d48-dd26r 1/1 Running 0 3d etc-release-etcd-operator-etcd-restore-operator-5cccd989c5snljc 1/1 Running 0 3d etcd-global-g8spqxw9ck 1/1 Running 0 1m etcd-global-jk9gnbbvsr 1/1 Running 0 1m etcd-global-mqttbs4555 1/1 Running 0 1m etcd-test-5hbbjzm9dx 1/1 Running 0 3d etcd-test-9d4k865bw8 1/1 Running 0 3d etcd-test-lvdkrt4jtq 1/1 Running 0 3d etcd-zone1-8l2r99dj8g 1/1 Running 0 1m etcd-zone1-gvrpqqg9wg 1/1 Running 0 1m etcd-zone1-hwtwd7xch2 1/1 Running 0 1m ... vtctld-85b768b495-xsj4m 1/1 Running 0 1m vtgate-zone1-758bd8fbc8-2xsjn 0/1 CrashLoopBackOff 3 1m vtgate-zone1-758bd8fbc8-s9n6h 0/1 CrashLoopBackOff 3 1m vtgate-zone1-758bd8fbc8-vqhdc 0/1 CrashLoopBackOff 3 1m

root@tofu1:~/new/vitess/helm/vitess# kubectl describe pod vtgate-zone1-758bd8fbc8-2xsjn
Name:           vtgate-zone1-758bd8fbc8-2xsjn
Namespace:      default
Node:           minikube/9.202.176.72
Start Time:     Sat, 17 Nov 2018 04:18:54 -0800
Labels:         app=vitess
                cell=zone1
                component=vtgate
                pod-template-hash=3146849674
Annotations:    <none>
Status:         Running
IP:             172.17.0.69
Controlled By:  ReplicaSet/vtgate-zone1-758bd8fbc8
Containers:
  vtgate:
    Container ID:  docker://57ec5f45ff654a52ac6c558a8596cc90fc60d6990c15b1d747a62894e7de2f5b
    Image:         vitess/vtgate:latest
    Image ID:      docker-pullable://vitess/vtgate@sha256:ff6aaa5e2ecd4e26be27e33a96db2c5451c8c8275c3a6018ef757bb28cf276fc
    Port:          <none>
    Host Port:     <none>
    Command:
      bash
      -c
      set -ex

      eval exec /vt/bin/vtgate $(cat <<END_OF_COMMAND
        -topo_implementation=etcd2
        -topo_global_server_address="etcd-global-client.default:2379"
        -topo_global_root=/vitess/global
        -logtostderr=true
        -stderrthreshold=0
        -port=15001
        -grpc_port=15991

        -service_map="grpc-vtgateservice"
        -cells_to_watch="zone1"
        -tablet_types_to_wait="MASTER,REPLICA"
        -gateway_implementation="discoverygateway"
        -cell="zone1"

      END_OF_COMMAND
      )

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Sat, 17 Nov 2018 04:25:53 -0800
      Finished:     Sat, 17 Nov 2018 04:25:53 -0800
    Ready:          False
    Restart Count:  6
    Limits:
      cpu:     500m
      memory:  512Mi
    Requests:
      cpu:        500m
      memory:     512Mi
    Liveness:     http-get http://:15001/debug/status delay=30s timeout=5s period=10s #success=1 #failure=3
    Readiness:    http-get http://:15001/debug/health delay=30s timeout=5s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /mysqlcreds from creds (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-vvsxs (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  creds:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  default-token-vvsxs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-vvsxs
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                  From               Message
  ----     ------                 ----                 ----               -------
  Normal   Scheduled              10m                  default-scheduler  Successfully assigned vtgate-zone1-758bd8fbc8-2xsjn to minikube
  Normal   SuccessfulMountVolume  10m                  kubelet, minikube  MountVolume.SetUp succeeded for volume "creds"
  Normal   SuccessfulMountVolume  10m                  kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-vvsxs"
  Normal   Pulling                9m18s (x4 over 10m)  kubelet, minikube  pulling image "vitess/vtgate:latest"
  Normal   Pulled                 9m13s (x4 over 10m)  kubelet, minikube  Successfully pulled image "vitess/vtgate:latest"
  Normal   Created                9m13s (x4 over 10m)  kubelet, minikube  Created container
  Normal   Started                9m12s (x4 over 10m)  kubelet, minikube  Started container
  Warning  BackOff                41s (x47 over 10m)   kubelet, minikube  Back-off restarting failed container

Operating system and Environment details

Ubuntu 16.04

derekperkins commented 5 years ago

You need to add keyspace information in, otherwise no vttablets are deployed. The values.yaml file now shows an example, also you can reference the examples in the helm readme. https://github.com/vitessio/vitess/tree/master/helm/vitess