dragonflyoss / Dragonfly2

Dragonfly is an open source P2P-based file distribution and image acceleration system. It is hosted by the Cloud Native Computing Foundation (CNCF) as an Incubating Level Project.
https://d7y.io
Apache License 2.0
2.1k stars 264 forks source link

quick start issue with kind - mysql pod failed to run #3306

Open Idealist226 opened 1 month ago

Idealist226 commented 1 month ago

I followed the steps quick start to create a k8s cluster using kind.

But there was a problem when creating a dragonfly cluster using the provided configuration file charts-config.yaml:

$ kubectl get po -n dragonfly-system
NAME                                READY   STATUS     RESTARTS       AGE
dragonfly-dfdaemon-bph2z            0/1     Init:0/3   0              15m
dragonfly-dfdaemon-khw22            0/1     Init:0/3   0              15m
dragonfly-jaeger-66898d8bdb-hnh8v   1/1     Running    0              15m
dragonfly-manager-ccc657494-fwlzh   0/1     Init:1/2   0              15m
dragonfly-mysql-0                   0/1     Running    5 (112s ago)   15m
dragonfly-redis-master-0            1/1     Running    0              15m
dragonfly-redis-replicas-0          1/1     Running    1 (13m ago)    15m
dragonfly-redis-replicas-1          1/1     Running    0              13m
dragonfly-redis-replicas-2          1/1     Running    0              12m
dragonfly-scheduler-0               0/1     Init:0/1   0              15m
dragonfly-seed-peer-0               0/1     Init:0/1   0              15m

dfdaemon, scheduler, and manager are all in the init state because mysql did not run successfully.

$ kubectl describe pods dragonfly-mysql-0 -n dragonfly-system
Name:             dragonfly-mysql-0
Namespace:        dragonfly-system
Priority:         0
Service Account:  dragonfly-mysql
Node:             kind-worker/172.18.0.2
Start Time:       Wed, 05 Jun 2024 10:45:44 +0800
Labels:           app.kubernetes.io/component=primary
                  app.kubernetes.io/instance=dragonfly
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=mysql
                  apps.kubernetes.io/pod-index=0
                  controller-revision-hash=dragonfly-mysql-7cddcb867b
                  helm.sh/chart=mysql-9.4.6
                  statefulset.kubernetes.io/pod-name=dragonfly-mysql-0
Annotations:      checksum/configuration: 90920bcb341563e623d1eb9bfd866de2344f452610722a64ac0a0caa2e61e472
Status:           Running
IP:               10.244.1.8
IPs:
  IP:           10.244.1.8
Controlled By:  StatefulSet/dragonfly-mysql
Containers:
  mysql:
    Container ID:   containerd://fbb8f3561261a4c373d0a1b6b3baa4f1a0556b4b7b55b0e4a7931ea4ae6dccde
    Image:          docker.io/bitnami/mysql:8.0.31-debian-11-r30
    Image ID:       docker.io/bitnami/mysql@sha256:90e4eff2f8e964a955058ad7ba841d80a562196f8dd71732483a2861a4cec796
    Port:           3306/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Wed, 05 Jun 2024 11:00:46 +0800
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Wed, 05 Jun 2024 10:58:26 +0800
      Finished:     Wed, 05 Jun 2024 11:00:45 +0800
    Ready:          False
    Restart Count:  6
    Liveness:       exec [/bin/bash -ec password_aux="${MYSQL_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=5s timeout=1s period=10s #success=1 #failure=3
    Readiness:  exec [/bin/bash -ec password_aux="${MYSQL_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=5s timeout=1s period=10s #success=1 #failure=3
    Startup:  exec [/bin/bash -ec password_aux="${MYSQL_ROOT_PASSWORD:-}"
if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then
    password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")
fi
mysqladmin status -uroot -p"${password_aux}"
] delay=15s timeout=1s period=10s #success=1 #failure=10
    Environment:
      BITNAMI_DEBUG:        false
      MYSQL_ROOT_PASSWORD:  <set to the key 'mysql-root-password' in secret 'dragonfly-mysql'>  Optional: false
      MYSQL_USER:           dragonfly
      MYSQL_PASSWORD:       <set to the key 'mysql-password' in secret 'dragonfly-mysql'>  Optional: false
      MYSQL_DATABASE:       manager
    Mounts:
      /bitnami/mysql from data (rw)
      /opt/bitnami/mysql/conf/my.cnf from config (rw,path="my.cnf")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fwqdk (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-dragonfly-mysql-0
    ReadOnly:   false
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      dragonfly-mysql
    Optional:  false
  kube-api-access-fwqdk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  17m                  default-scheduler  Successfully assigned dragonfly-system/dragonfly-mysql-0 to kind-worker
  Normal   Pulling    17m                  kubelet            Pulling image "docker.io/bitnami/mysql:8.0.31-debian-11-r30"
  Normal   Pulled     16m                  kubelet            Successfully pulled image "docker.io/bitnami/mysql:8.0.31-debian-11-r30" in 37.974s (56.747s including waiting). Image size: 154852206 bytes.
  Normal   Killing    14m                  kubelet            Container mysql failed startup probe, will be restarted
  Normal   Created    13m (x2 over 16m)    kubelet            Created container mysql
  Normal   Started    13m (x2 over 16m)    kubelet            Started container mysql
  Normal   Pulled     6m48s (x4 over 13m)  kubelet            Container image "docker.io/bitnami/mysql:8.0.31-debian-11-r30" already present on machine
  Warning  Unhealthy  108s (x61 over 15m)  kubelet            Startup probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.
mysqladmin: connect to server at 'localhost' failed
error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mysql/tmp/mysql.sock' (2)'
Check that mysqld is running and that the socket: '/opt/bitnami/mysql/tmp/mysql.sock' exists!

The environment I use are as follow:

gaius-qi commented 3 weeks ago

I followed the steps quick start to create a k8s cluster using kind.

But there was a problem when creating a dragonfly cluster using the provided configuration file charts-config.yaml:


$ kubectl get po -n dragonfly-system

NAME                                READY   STATUS     RESTARTS       AGE

dragonfly-dfdaemon-bph2z            0/1     Init:0/3   0              15m

dragonfly-dfdaemon-khw22            0/1     Init:0/3   0              15m

dragonfly-jaeger-66898d8bdb-hnh8v   1/1     Running    0              15m

dragonfly-manager-ccc657494-fwlzh   0/1     Init:1/2   0              15m

dragonfly-mysql-0                   0/1     Running    5 (112s ago)   15m

dragonfly-redis-master-0            1/1     Running    0              15m

dragonfly-redis-replicas-0          1/1     Running    1 (13m ago)    15m

dragonfly-redis-replicas-1          1/1     Running    0              13m

dragonfly-redis-replicas-2          1/1     Running    0              12m

dragonfly-scheduler-0               0/1     Init:0/1   0              15m

dragonfly-seed-peer-0               0/1     Init:0/1   0              15m

dfdaemon, scheduler, and manager are all in the init state because mysql did not run successfully.


$ kubectl describe pods dragonfly-mysql-0 -n dragonfly-system

Name:             dragonfly-mysql-0

Namespace:        dragonfly-system

Priority:         0

Service Account:  dragonfly-mysql

Node:             kind-worker/172.18.0.2

Start Time:       Wed, 05 Jun 2024 10:45:44 +0800

Labels:           app.kubernetes.io/component=primary

                  app.kubernetes.io/instance=dragonfly

                  app.kubernetes.io/managed-by=Helm

                  app.kubernetes.io/name=mysql

                  apps.kubernetes.io/pod-index=0

                  controller-revision-hash=dragonfly-mysql-7cddcb867b

                  helm.sh/chart=mysql-9.4.6

                  statefulset.kubernetes.io/pod-name=dragonfly-mysql-0

Annotations:      checksum/configuration: 90920bcb341563e623d1eb9bfd866de2344f452610722a64ac0a0caa2e61e472

Status:           Running

IP:               10.244.1.8

IPs:

  IP:           10.244.1.8

Controlled By:  StatefulSet/dragonfly-mysql

Containers:

  mysql:

    Container ID:   containerd://fbb8f3561261a4c373d0a1b6b3baa4f1a0556b4b7b55b0e4a7931ea4ae6dccde

    Image:          docker.io/bitnami/mysql:8.0.31-debian-11-r30

    Image ID:       docker.io/bitnami/mysql@sha256:90e4eff2f8e964a955058ad7ba841d80a562196f8dd71732483a2861a4cec796

    Port:           3306/TCP

    Host Port:      0/TCP

    State:          Running

      Started:      Wed, 05 Jun 2024 11:00:46 +0800

    Last State:     Terminated

      Reason:       Error

      Exit Code:    137

      Started:      Wed, 05 Jun 2024 10:58:26 +0800

      Finished:     Wed, 05 Jun 2024 11:00:45 +0800

    Ready:          False

    Restart Count:  6

    Liveness:       exec [/bin/bash -ec password_aux="${MYSQL_ROOT_PASSWORD:-}"

if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then

    password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")

fi

mysqladmin status -uroot -p"${password_aux}"

] delay=5s timeout=1s period=10s #success=1 #failure=3

    Readiness:  exec [/bin/bash -ec password_aux="${MYSQL_ROOT_PASSWORD:-}"

if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then

    password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")

fi

mysqladmin status -uroot -p"${password_aux}"

] delay=5s timeout=1s period=10s #success=1 #failure=3

    Startup:  exec [/bin/bash -ec password_aux="${MYSQL_ROOT_PASSWORD:-}"

if [[ -f "${MYSQL_ROOT_PASSWORD_FILE:-}" ]]; then

    password_aux=$(cat "$MYSQL_ROOT_PASSWORD_FILE")

fi

mysqladmin status -uroot -p"${password_aux}"

] delay=15s timeout=1s period=10s #success=1 #failure=10

    Environment:

      BITNAMI_DEBUG:        false

      MYSQL_ROOT_PASSWORD:  <set to the key 'mysql-root-password' in secret 'dragonfly-mysql'>  Optional: false

      MYSQL_USER:           dragonfly

      MYSQL_PASSWORD:       <set to the key 'mysql-password' in secret 'dragonfly-mysql'>  Optional: false

      MYSQL_DATABASE:       manager

    Mounts:

      /bitnami/mysql from data (rw)

      /opt/bitnami/mysql/conf/my.cnf from config (rw,path="my.cnf")

      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fwqdk (ro)

Conditions:

  Type                        Status

  PodReadyToStartContainers   True

  Initialized                 True

  Ready                       False

  ContainersReady             False

  PodScheduled                True

Volumes:

  data:

    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

    ClaimName:  data-dragonfly-mysql-0

    ReadOnly:   false

  config:

    Type:      ConfigMap (a volume populated by a ConfigMap)

    Name:      dragonfly-mysql

    Optional:  false

  kube-api-access-fwqdk:

    Type:                    Projected (a volume that contains injected data from multiple sources)

    TokenExpirationSeconds:  3607

    ConfigMapName:           kube-root-ca.crt

    ConfigMapOptional:       <nil>

    DownwardAPI:             true

QoS Class:                   BestEffort

Node-Selectors:              <none>

Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s

                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:

  Type     Reason     Age                  From               Message

  ----     ------     ----                 ----               -------

  Normal   Scheduled  17m                  default-scheduler  Successfully assigned dragonfly-system/dragonfly-mysql-0 to kind-worker

  Normal   Pulling    17m                  kubelet            Pulling image "docker.io/bitnami/mysql:8.0.31-debian-11-r30"

  Normal   Pulled     16m                  kubelet            Successfully pulled image "docker.io/bitnami/mysql:8.0.31-debian-11-r30" in 37.974s (56.747s including waiting). Image size: 154852206 bytes.

  Normal   Killing    14m                  kubelet            Container mysql failed startup probe, will be restarted

  Normal   Created    13m (x2 over 16m)    kubelet            Created container mysql

  Normal   Started    13m (x2 over 16m)    kubelet            Started container mysql

  Normal   Pulled     6m48s (x4 over 13m)  kubelet            Container image "docker.io/bitnami/mysql:8.0.31-debian-11-r30" already present on machine

  Warning  Unhealthy  108s (x61 over 15m)  kubelet            Startup probe failed: mysqladmin: [Warning] Using a password on the command line interface can be insecure.

mysqladmin: connect to server at 'localhost' failed

error: 'Can't connect to local MySQL server through socket '/opt/bitnami/mysql/tmp/mysql.sock' (2)'

Check that mysqld is running and that the socket: '/opt/bitnami/mysql/tmp/mysql.sock' exists!

The environment I use are as follow:

  • OS: wsl2 Ubuntu 20.04

  • Kernel: 5.15.146.1-microsoft-standard-WSL2

  • kind: v0.23.0 go1.21.10 linux/amd64

  • helm: version.BuildInfo{Version:"v3.15.1", GitCommit:"e211f2aa62992bd72586b395de50979e31231829", GitTreeState:"clean", GoVersion:"go1.22.3"}

Dragonfly chart uses official mysql chart. If you have problem about mysql chart, please contact MySQL community.