search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.08k stars 170 forks source link

[BUG] apecloud-mysql cluster hscale 1 to 3 pod CrashLoopBackOff after bench #6562

Closed JashBook closed 3 months ago

JashBook commented 8 months ago

Describe the bug apecloud-mysql cluster hscale 1 to 3 pod CrashLoopBackOff after bench.

kbcli version
Kubernetes: v1.26.6-gke.1700
KubeBlocks: 0.8.2-beta.1
kbcli: 0.8.1

To Reproduce Steps to reproduce the behavior:

  1. create apecloud-mysql cluster 
    `kbcli cluster create  mysql-hkampm --termination-policy=Halt --monitoring-interval=0 --cluster-definition=apecloud-mysql --enable-all-logs=false --cluster-version=ac-mysql-8.0.30 --set cpu=100m,memory=0.5Gi,replicas=1,storage=20Gi  --namespace default
  2. kbcli bench 
    kbcli bench  sysbench benchdb-mysql-hkampm --cluster mysql-hkampm --host mysql-hkampm-mysql.default.svc --user root --password "m7fnm8p9" --database benchdb  --namespace default --port 3306 --threads 10                     --type oltp_read_write --tables 10 --size 20000 --duration 120 --driver mysql                      --extra-args="--auto_inc=false"
  3. hscale 1 to 3 
    kbcli cluster hscale mysql-hkampm --auto-approve --components mysql --replicas 3 --namespace default
  4. See error 
    
kubectl get all -l app.kubernetes.io/instance=mysql-hkampm

NAME READY STATUS RESTARTS AGE pod/mysql-hkampm-mysql-0 5/5 Running 0 30m pod/mysql-hkampm-mysql-1 4/5 CrashLoopBackOff 9 (36s ago) 24m pod/mysql-hkampm-mysql-2 4/5 CrashLoopBackOff 9 (38s ago) 24m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/mysql-hkampm-mysql ClusterIP 10.132.12.73 3306/TCP 30m service/mysql-hkampm-mysql-headless ClusterIP None

NAME READY AGE statefulset.apps/mysql-hkampm-mysql 1/3 30m

NAME CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS AGE cluster.apps.kubeblocks.io/mysql-hkampm apecloud-mysql ac-mysql-8.0.30 WipeOut Failed 30m

NAME COMPONENT-DEFINITION STATUS AGE component.apps.kubeblocks.io/mysql-hkampm-mysql Failed 30m

NAME LEADER READY REPLICAS AGE replicatedstatemachine.workloads.kubeblocks.io/mysql-hkampm-mysql 1 3 30m

describe pod

kubectl describe pod mysql-hkampm-mysql-1 Name: mysql-hkampm-mysql-1 Namespace: default Priority: 0 Node: gke-infracreate-gke-default-pool-61c5a7f3-wbwm/10.10.0.67 Start Time: Wed, 31 Jan 2024 11:05:54 +0800 Labels: app.kubernetes.io/component=mysql app.kubernetes.io/instance=mysql-hkampm app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=apecloud-mysql app.kubernetes.io/version= apps.kubeblocks.io/component-name=mysql controller-revision-hash=mysql-hkampm-mysql-7885f86fbc statefulset.kubernetes.io/pod-name=mysql-hkampm-mysql-1 Annotations: apps.kubeblocks.io/component-replicas: 3 Status: Running IP: 10.128.6.60 IPs: IP: 10.128.6.60 Controlled By: StatefulSet/mysql-hkampm-mysql Containers: mysql: Container ID: containerd://88b8c4b29ba050ad5ea9020856616eb9bb1243766513c0fc48db498b3a91f6dc Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta3.20231215.ge77d836.13 Image ID: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server@sha256:1b98d61f6f61e288dca34eb6ad53ecae2c113903e73a7e4ebb955a6ed19b9f33 Ports: 3306/TCP, 13306/TCP Host Ports: 0/TCP, 0/TCP Command: /scripts/setup.sh State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: Error Exit Code: 1 Started: Wed, 31 Jan 2024 11:29:42 +0800 Finished: Wed, 31 Jan 2024 11:29:43 +0800 Ready: False Restart Count: 9 Limits: cpu: 100m memory: 512Mi Requests: cpu: 100m memory: 512Mi Environment Variables from: mysql-hkampm-mysql-env ConfigMap Optional: false mysql-hkampm-mysql-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: mysql-hkampm-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).mysql-hkampm-mysql-headless.$(KB_NAMESPACE).svc SERVICE_PORT: 3306 MYSQL_ROOT_HOST: % MYSQL_ROOT_USER: <set to the key 'username' in secret 'mysql-hkampm-conn-credential'> Optional: false MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'mysql-hkampm-conn-credential'> Optional: false MYSQL_DATABASE: mydb MYSQL_USER: u1 MYSQL_PASSWORD: u1 CLUSTER_ID: 1 CLUSTER_START_INDEX: 1 REPLICATION_USER: replicator REPLICATION_PASSWORD:
MYSQL_TEMPLATE_CONFIG:
MYSQL_CUSTOM_CONFIG:
MYSQL_DYNAMIC_CONFIG:
KB_EMBEDDED_WESQL: 1 Mounts: /data/mysql from data (rw) /etc/annotations from annotations (rw) /opt/mysql from mysql-config (rw) /scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b2w9g (ro) metrics: Container ID: containerd://0e6421fe45f6316efd40df04223fbdedf6120d9b1681b035849702833c154e93 Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1 Image ID: docker.io/apecloud/agamotto@sha256:cbab349b90490807a8d5039bf01bc7e37334f20c98c7dd75bc7fc4cf9e5b10ee Port: 9104/TCP Host Port: 0/TCP Command: /scripts/agamotto.sh State: Running Started: Wed, 31 Jan 2024 11:06:01 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: mysql-hkampm-mysql-env ConfigMap Optional: false mysql-hkampm-mysql-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: mysql-hkampm-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).mysql-hkampm-mysql-headless.$(KB_NAMESPACE).svc DB_TYPE: MySQL ENDPOINT: localhost:3306 MYSQL_USER: <set to the key 'username' in secret 'mysql-hkampm-conn-credential'> Optional: false MYSQL_PASSWORD: <set to the key 'password' in secret 'mysql-hkampm-conn-credential'> Optional: false Mounts: /data/mysql from data (rw) /opt/agamotto from agamotto-configuration (rw) /scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b2w9g (ro) vttablet: Container ID: containerd://0dd526c6b3cffe75e03daadf64f4e2006854a362ecc530f044094db62442094c Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale:0.2.3 Image ID: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale@sha256:d4ac8323dcaa50e24f07df68017cccdb9511e0deaf642a7f4755b2b4d448df4d Ports: 15100/TCP, 16100/TCP Host Ports: 0/TCP, 0/TCP Command: /scripts/vttablet.sh State: Running Started: Wed, 31 Jan 2024 11:06:02 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: mysql-hkampm-mysql-env ConfigMap Optional: false mysql-hkampm-mysql-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: mysql-hkampm-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).mysql-hkampm-mysql-headless.$(KB_NAMESPACE).svc CELL: zone1 ETCD_SERVER: $(KB_CLUSTER_NAME)-vtcontroller-headless ETCD_PORT: 2379 TOPOLOGY_FLAGS: --topo_implementation etcd2 --topo_global_server_address $(ETCD_SERVER):$(ETCD_PORT) --topo_global_root /vitess/global VTTABLET_PORT: 15100 VTTABLET_GRPC_PORT: 16100 VTCTLD_HOST: $(KB_CLUSTER_NAME)-vtcontroller-headless VTCTLD_WEB_PORT: 15000 SERVICE_PORT: $(VTTABLET_PORT) MYSQL_ROOT_USER: <set to the key 'username' in secret 'mysql-hkampm-conn-credential'> Optional: false MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'mysql-hkampm-conn-credential'> Optional: false Mounts: /conf from mysql-scale-config (rw) /scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b2w9g (ro) /vtdataroot from data (rw) kb-checkrole: Container ID: containerd://70a81478b977ac91d6dfa43731b5b125659bb51eb4efb216938ae65a6b6d1a2f Image: docker.io/apecloud/kubeblocks-tools:0.8.2-beta.1 Image ID: docker.io/apecloud/kubeblocks-tools@sha256:ea6ccdbbbe9a629e6558550fe84709cb41749b752490ec5a6a2b844297715251 Ports: 3501/TCP, 50001/TCP Host Ports: 0/TCP, 0/TCP Command: lorry --port 3501 --grpcport 50001 State: Running Started: Wed, 31 Jan 2024 11:06:02 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Readiness: http-get http://:3501/v1.0/checkrole delay=0s timeout=1s period=1s #success=1 #failure=2 Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: mysql-hkampm-mysql-env ConfigMap Optional: false mysql-hkampm-mysql-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: mysql-hkampm-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).mysql-hkampm-mysql-headless.$(KB_NAMESPACE).svc KB_SERVICE_PORT: 3306 KB_DATA_PATH: /data/mysql KB_BUILTIN_HANDLER: wesql KB_SERVICE_USER: <set to the key 'username' in secret 'mysql-hkampm-conn-credential'> Optional: false KB_SERVICE_PASSWORD: <set to the key 'password' in secret 'mysql-hkampm-conn-credential'> Optional: false KB_RSM_ACTION_SVC_LIST: null KB_RSM_ROLE_UPDATE_MECHANISM: DirectAPIServerEventUpdate KB_RSM_ROLE_PROBE_TIMEOUT: 1 KB_CLUSTER_NAME: (v1:metadata.labels['app.kubernetes.io/instance']) KB_COMP_NAME: (v1:metadata.labels['apps.kubeblocks.io/component-name']) KB_SERVICE_CHARACTER_TYPE: wesql Mounts: /data/mysql from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b2w9g (ro) config-manager: Container ID: containerd://aec172b6217e0cb4f1b572d16d11731862e07d718725ccafaef7c50f95865f71 Image: docker.io/apecloud/kubeblocks-tools:0.8.2-beta.1 Image ID: docker.io/apecloud/kubeblocks-tools@sha256:ea6ccdbbbe9a629e6558550fe84709cb41749b752490ec5a6a2b844297715251 Port: Host Port: Command: env Args: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH) /bin/reloader --log-level info --volume-dir /conf --operator-update-enable --tcp 9901 --config /opt/config-manager/config-manager.yaml State: Running Started: Wed, 31 Jan 2024 11:06:02 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: mysql-hkampm-mysql-env ConfigMap Optional: false mysql-hkampm-mysql-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: mysql-hkampm-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).mysql-hkampm-mysql-headless.$(KB_NAMESPACE).svc CONFIG_MANAGER_POD_IP: (v1:status.podIP) DB_TYPE: mysql MYSQL_USER: <set to the key 'username' in secret 'mysql-hkampm-conn-credential'> Optional: false MYSQL_PASSWORD: <set to the key 'password' in secret 'mysql-hkampm-conn-credential'> Optional: false DATA_SOURCE_NAME: $(MYSQL_USER):$(MYSQL_PASSWORD)@(localhost:3306)/ TOOLS_PATH: /opt/kb-tools/reload/mysql-consensusset-config:/opt/config-manager Mounts: /conf from mysql-scale-config (rw) /opt/config-manager from config-manager-config (rw) /opt/kb-tools/reload/mysql-consensusset-config from cm-script-mysql-consensusset-config (rw) /opt/mysql from mysql-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b2w9g (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-mysql-hkampm-mysql-1 ReadOnly: false annotations: Type: DownwardAPI (a volume populated by information about the pod) Items: metadata.annotations['cs.apps.kubeblocks.io/leader'] -> leader metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas agamotto-configuration: Type: ConfigMap (a volume populated by a ConfigMap) Name: mysql-hkampm-mysql-agamotto-configuration Optional: false scripts: Type: ConfigMap (a volume populated by a ConfigMap) Name: mysql-hkampm-mysql-apecloud-mysql-scripts Optional: false mysql-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: mysql-hkampm-mysql-mysql-consensusset-config Optional: false mysql-scale-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: mysql-hkampm-mysql-vttablet-config Optional: false cm-script-mysql-consensusset-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-mysql-reload-script-mysql-hkampm Optional: false config-manager-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-mysql-hkampm-mysql-config-manager-config Optional: false kube-api-access-b2w9g: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: kb-data=true:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal Scheduled 26m default-scheduler Successfully assigned default/mysql-hkampm-mysql-1 to gke-infracreate-gke-default-pool-61c5a7f3-wbwm Normal Created 26m kubelet Created container metrics Normal Pulled 26m kubelet Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale:0.2.3" already present on machine Normal Started 26m kubelet Started container metrics Normal Pulled 26m kubelet Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1" already present on machine Normal Created 26m kubelet Created container vttablet Normal Started 26m kubelet Started container kb-checkrole Normal Pulled 26m kubelet Container image "docker.io/apecloud/kubeblocks-tools:0.8.2-beta.1" already present on machine Normal Created 26m kubelet Created container config-manager Normal Started 26m kubelet Started container vttablet Normal Pulled 26m kubelet Container image "docker.io/apecloud/kubeblocks-tools:0.8.2-beta.1" already present on machine Normal Created 26m kubelet Created container kb-checkrole Normal Started 26m kubelet Started container config-manager Normal Started 23m (x3 over 26m) kubelet Started container mysql Normal Created 23m (x3 over 26m) kubelet Created container mysql Normal Pulled 22m (x4 over 26m) kubelet Container image "infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta3.20231215.ge77d836.13" already present on machine Warning BackOff 62s (x104 over 23m) kubelet Back-off restarting failed container mysql in pod mysql-hkampm-mysql-1_default(78b4f381-66bd-41f4-bd33-805585389595)

logs pod

➜ ~ kubectl logs mysql-hkampm-mysql-1 mysql 2024-01-31 03:29:43+00:00 [Note] [MySQL]: Entrypoint script for MySQL Server started. 2024-01-31 03:29:43+00:00 [Note] [MySQL]: KB_EMBEDDED_WESQL=1 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: Welcome to the KubeBlocks MySQL Cluster Container 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: KB_VOLUME_DIR=/data/mysql 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: KB_DATA_DIR=/data/mysql/data 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: KB_LOG_DIR=/data/mysql/log 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: KB_CLUSTER_LEADER= 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: KB_CLUSTER_FOLLOWERS= 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: KB_CLUSTER_UID=43898ac9-5d58-4383-9a24-6a70ca6cfb3f 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: KB_CLUSTER_UID_PATH=/data/mysql/.kb_cluster_uid 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: KB_MYSQL_CONF_FILE=/opt/mysql/my.cnf 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: KB_MYSQL_PORT=3306 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: KB_POD_NAME=mysql-hkampm-mysql-1 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: KB_POD_N=3 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: KB_POD_INDEX=1 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: KB_POD_HOST=mysql-hkampm-mysql-1.mysql-hkampm-mysql-headless 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: 2024-01-31 03:29:43+00:00 [Note] [KubeBlocks]: Found restore files in /data/mysql/data /apecloud/mysql/scripts/libkb.sh: line 63: KB_CLUSTER_LEADER: missing leader ➜ ~ ➜ ~ kubectl logs mysql-hkampm-mysql-2 mysql 2024-01-31 03:29:40+00:00 [Note] [MySQL]: Entrypoint script for MySQL Server started. 2024-01-31 03:29:41+00:00 [Note] [MySQL]: KB_EMBEDDED_WESQL=1 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: Welcome to the KubeBlocks MySQL Cluster Container 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: KB_VOLUME_DIR=/data/mysql 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: KB_DATA_DIR=/data/mysql/data 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: KB_LOG_DIR=/data/mysql/log 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: KB_CLUSTER_LEADER= 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: KB_CLUSTER_FOLLOWERS= 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: KB_CLUSTER_UID=43898ac9-5d58-4383-9a24-6a70ca6cfb3f 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: KB_CLUSTER_UID_PATH=/data/mysql/.kb_cluster_uid 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: KB_MYSQL_CONF_FILE=/opt/mysql/my.cnf 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: KB_MYSQL_PORT=3306 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: KB_POD_NAME=mysql-hkampm-mysql-2 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: KB_POD_N=3 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: KB_POD_INDEX=2 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: KB_POD_HOST=mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: 2024-01-31 03:29:41+00:00 [Note] [KubeBlocks]: Found restore files in /data/mysql/data /apecloud/mysql/scripts/libkb.sh: line 63: KB_CLUSTER_LEADER: missing leader

mysql error logs

kubectl exec -it mysql-hkampm-mysql-1 -c metrics sh
/ # cat data/mysql/log/mysqld-error.log 2024-01-31T03:00:06.350346Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_exec_mode instead. 2024-01-31T03:00:06.353164Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.30) initializing of server in progress as process 277 2024-01-31T03:00:06.356372Z 0 [Warning] [MY-010161] [Server] You need to use --log-bin to make --log-replica-updates work. 2024-01-31T03:00:06.356384Z 0 [Warning] [MY-010161] [Server] You need to use --log-bin to make --binlog-format work. 2024-01-31T03:00:06.466845Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2024-01-31T03:00:08.376136Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2024-01-31T03:00:18.148649Z 0 [Warning] [MY-010161] [Server] You need to use --log-bin to make --binlog-expire-logs-seconds work. 2024-01-31T03:00:25.853429Z 5 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option. 2024-01-31T03:00:46.150515Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_exec_mode instead. 2024-01-31T03:00:46.151050Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 325 2024-01-31T03:00:46.250672Z 0 [Warning] [MY-010161] [Server] You need to use --log-bin to make --log-replica-updates work. 2024-01-31T03:00:46.250684Z 0 [Warning] [MY-010161] [Server] You need to use --log-bin to make --binlog-format work. 2024-01-31T03:00:46.352619Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2024-01-31T03:00:48.366803Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2024-01-31T03:00:52.552165Z 0 [Warning] [MY-010161] [Server] You need to use --log-bin to make --binlog-expire-logs-seconds work. 2024-01-31T03:00:52.553994Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. 2024-01-31T03:00:52.554073Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel. 2024-01-31T03:00:52.753586Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30' socket: '/var/run/mysqld/mysqld.sock' port: 0 WeSQL Server - GPL, Release 5, Revision e77d836. 2024-01-31T03:01:27.959699Z 12 [System] [MY-013172] [Server] Received SHUTDOWN from user root. Shutting down mysqld (Version: 8.0.30). 2024-01-31T03:01:29.260813Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30) WeSQL Server - GPL, Release 5, Revision e77d836. 2024-01-31T03:01:30.547378Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_exec_mode instead. 2024-01-31T03:01:30.547861Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 430 2024-01-31T03:01:30.647033Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2024-01-31T03:01:32.458235Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2024-01-31T03:01:34.448184Z 4 [System] [MY-013381] [Server] Server upgrade from '80030' to '80030' started. 2024-01-31T03:01:34.951848Z 4 [System] [MY-013381] [Server] Server upgrade from '80030' to '80030' completed. 2024-01-31T03:01:35.851844Z 0 [Warning] [MY-000000] [Server] Force change meta to system table successfully. 2024-01-31T03:01:36.559872Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30) WeSQL Server - GPL, Release 5, Revision e77d836. 2024-01-31T03:01:37.948204Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_exec_mode instead. 2024-01-31T03:01:37.948962Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 7 2024-01-31T03:01:38.049762Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2024-01-31T03:01:39.948808Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2024-01-31T03:01:41.859802Z 0 [System] [MY-010229] [Server] Starting XA crash recovery... 2024-01-31T03:01:42.152922Z 0 [System] [MY-010232] [Server] XA crash recovery finished. [2024-01-31 03:01:43.832221] [Default] Server 1 : Enter startElectionCallback [2024-01-31 03:01:43.832221] [Default] Server 1 : Paxos state change from FOLL to CAND !! [2024-01-31 03:01:43.832221] [Default] Server 1 : Epoch task currentEpoch(0) during requestVote [2024-01-31 03:01:43.832221] [Default] Server 1 : Start new requestVote: new term(2) [2024-01-31 03:01:43.832221] [Default] Server 1 : Paxos state change from CAND to LEDR !! [2024-01-31 03:01:43.832221] [Default] Server 1 : become Leader (currentTerm 2, lli:1, llt:2)!! 2024-01-31T03:01:44.052103Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. 2024-01-31T03:01:44.052162Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel. 2024-01-31T03:01:44.653012Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30' socket: '/var/run/mysqld/mysqld.sock' port: 3306 WeSQL Server - GPL, Release 5, Revision e77d836. 2024-01-31T03:01:44.748473Z 6 [System] [MY-025003] [Server] Consensus apply thread start, recover status = 0, consensus start apply index = 0, rli consensus index = 0. 2024-01-31T03:01:44.748602Z 6 [System] [MY-025004] [Server] Consensus apply thread group relay log file name = '/data/mysql/binlog/mysql-bin.000001', pos = 251, rli apply index = 0. 2024-01-31T03:01:44.754745Z 6 [System] [MY-025005] [Server] Consensus apply thread stop, opt_consensus_leader_stop_apply: false, seconds_behind_master: 1706670104, opt_consensus_leader_stop_apply_time: 0. 2024-01-31T03:01:44.754791Z 6 [System] [MY-025006] [Server] Consensus apply thread catchup commit index, consensus index: 0, current term: 1, stop term: 2. 2024-01-31T03:06:15.455712Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_exec_mode instead. 2024-01-31T03:06:15.457890Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 286 2024-01-31T03:06:15.557187Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2024-01-31T03:06:23.257700Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2024-01-31T03:06:25.259507Z 4 [System] [MY-013381] [Server] Server upgrade from '80030' to '80030' started. 2024-01-31T03:06:25.461909Z 4 [System] [MY-013381] [Server] Server upgrade from '80030' to '80030' completed. 2024-01-31T03:06:27.873406Z 0 [System] [MY-010229] [Server] Starting XA crash recovery... 2024-01-31T03:06:28.356806Z 0 [System] [MY-010232] [Server] XA crash recovery finished. 2024-01-31T03:06:28.965479Z 0 [Warning] [MY-000000] [Server] Force change meta to system table successfully. 2024-01-31T03:06:30.464474Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30) WeSQL Server - GPL, Release 5, Revision e77d836. 2024-01-31T03:06:31.458385Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_execmode instead. 2024-01-31T03:06:31.458950Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.30) starting as process 328 2024-01-31T03:06:31.657314Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started. 2024-01-31T03:06:36.559014Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended. 2024-01-31T03:06:41.857839Z 0 [System] [MY-010229] [Server] Starting XA crash recovery... 2024-01-31T03:06:42.058441Z 0 [System] [MY-010232] [Server] XA crash recovery finished. [2024-01-31 03:06:43.758594] [Default] Start init node as a learner. [2024-01-31 03:06:43.964169] [Default] Server 101 : applyConfigureChange done! logIndex(1698) currentTerm(2) val.cctype(2) val.optype(3) [2024-01-31 03:06:43.964169] [Default] Server 101 : applyConfigureChange_ done! logIndex(1699) currentTerm(2) val.cctype(2) val.optype(3) 2024-01-31T03:06:45.560652Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed. 2024-01-31T03:06:45.560708Z 0 [System] [MY-013602] [Server] Channel mysqlmain configured to support TLS. Encrypted connections are now supported for this channel. 2024-01-31T03:06:46.256371Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.30' socket: '/var/run/mysqld/mysqld.sock' port: 0 WeSQL Server - GPL, Release 5, Revision e77d836. 2024-01-31T03:06:46.362453Z 6 [System] [MY-025003] [Server] Consensus apply thread start, recover status = 1, consensus start apply index = 0, rli consensus index = 0. 2024-01-31T03:06:46.362645Z 6 [System] [MY-025004] [Server] Consensus apply thread group relay log file name = '/data/mysql/binlog/mysql-bin.000001', pos = 40595718, rli apply index = 648. [2024-01-31 03:06:46.867667] [ERROR] Server 101 : the server id in the msg(2) is not match with local server id for a learner, we change the local server id to 2!! [2024-01-31 03:06:46.867667] [Default] Server 2 : Paxos state change from LENR to FOLL !! [2024-01-31 03:06:47.059236] [Default] EasyNet::onConnected server 1 to port 13306 [2024-01-31 03:06:46.867667] [Default] Server 2 : Init follower from learner, new members(mysql-hkampm-mysql-0.mysql-hkampm-mysql-headless:13306#5N mysql-hkampm-mysql-1.mysql-hkampm-mysql-headless:13306 ) new learners(mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306$00 0 ) [2024-01-31 03:06:46.867667] [Default] Server 2 : applyConfigureChange done! logIndex(1706) currentTerm(2) val.cctype(1) val.optype(3) [2024-01-31 03:06:49.355374] [Default] Server 2 : leaderStickiness check: msg::force(0) state:0 electionTimer::Stage:1 leaderId_:1 . [2024-01-31 03:06:49.355374] [Default] Server 2 : New Term in onRequestVote !! server 1 's term(4) is bigger than me(2). [2024-01-31 03:06:49.355374] [Default] Server 2 : new term(old:2,new:4) !! [2024-01-31 03:06:49.355374] [Default] Server 2 : Paxos state change from FOLL to FOLL !! [2024-01-31 03:06:49.555363] [Default] Server 2 : Enter startElectionCallback [2024-01-31 03:06:49.355374] [Default] Server 2 : isVote: 1, local(lli:1706, llt:2); msg(candidateid: 1, term: 4 lli:1706, llt:2) . [2024-01-31 03:06:49.555363] [Default] Server 2 : Paxos state change from FOLL to CAND !! [2024-01-31 03:06:49.555363] [Default] Server 2 : Epoch task currentEpoch(0) during requestVote [2024-01-31 03:06:49.555363] [Default] Server 2 : Start new requestVote: new term(5) [2024-01-31 03:06:50.355384] [Default] Server 2 : server 1 (term:5) vote me to became leader. [2024-01-31 03:06:50.355384] [Default] Server 2 : Paxos state change from CAND to LEDR !! [2024-01-31 03:06:50.355384] [Default] Server 2 : become Leader (currentTerm 5, lli:1707, llt:5)!! [2024-01-31 03:06:50.363687] [Default] EasyNet::onConnected server 100 to port 13306 [2024-01-31 03:06:50.363687] [Default] Server 2 : update server 100 's nextIndex(old:1707,new:1706) when onConnect [2024-01-31 03:06:50.656125] [Default] EasyNet::onDisconnected server 100 [2024-01-31 03:06:50.706692] [Default] EasyNet::onConnected server 100 to port 13306 [2024-01-31 03:06:50.706692] [Default] Server 2 : update server 100 's nextIndex(old:1706,new:1706) when onConnect [2024-01-31 03:06:54.756225] [Default] EasyNet::onDisconnected server 100 [2024-01-31 03:06:54.806668] [Default] EasyNet::onConnected server 100 to port 13306 [2024-01-31 03:06:54.806668] [Default] Server 2 : update server 100 's nextIndex(old:1708,new:1708) when onConnect [2024-01-31 03:07:01.457747] [Default] EasyNet::onDisconnected server 100 [2024-01-31 03:07:01.555557] [Default] EasyNet::onConnected server 100 to port 13306 [2024-01-31 03:07:01.555557] [Default] Server 2 : update server 100 's nextIndex(old:1708,new:1708) when onConnect 2024-01-31T03:07:01.558684Z 0 [Warning] [MY-010382] [Repl] The transaction owned GTID is already in the gtid_executed table, which is caused by an explicit modifying from user client. 2024-01-31T03:07:01.558749Z 0 [ERROR] [MY-025008] [Repl] [GTID INFO] The transaction owned GTID gno start 1, end 647. [2024-01-31 03:07:02.757698] [Default] EasyNet::onDisconnected server 100 [2024-01-31 03:07:02.855530] [Default] EasyNet::onConnected server 100 to port 13306 [2024-01-31 03:07:02.855530] [Default] Server 2 : update server 100 's nextIndex(old:1708,new:1708) when onConnect [2024-01-31 03:07:23.455713] [Default] EasyNet::onDisconnected server 100 [2024-01-31 03:07:23.506133] [Default] EasyNet::onConnected server 100 to port 13306 [2024-01-31 03:07:23.506133] [Default] Server 2 : update server 100 's nextIndex(old:1708,new:1708) when onConnect [2024-01-31 03:07:37.355673] [Default] EasyNet::onDisconnected server 100 [2024-01-31 03:07:37.406141] [Default] EasyNet::onConnected server 100 to port 13306 [2024-01-31 03:07:37.406141] [Default] Server 2 : update server 100 's nextIndex(old:1708,new:1708) when onConnect [2024-01-31 03:07:48.455804] [Default] EasyNet::onDisconnected server 100 [2024-01-31 03:07:48.555546] [Default] EasyNet::onConnected server 100 to port 13306 [2024-01-31 03:07:48.555546] [Default] Server 2 : update server 100 's nextIndex(old:1708,new:1708) when onConnect [2024-01-31 03:07:54.755356] [Default] EasyNet::onDisconnected server 100 [2024-01-31 03:07:54.806045] [Default] EasyNet::onConnected server 100 to port 13306 [2024-01-31 03:07:54.806045] [Default] Server 2 : update server 100 's nextIndex(old:1708,new:1708) when onConnect [2024-01-31 03:08:14.155534] [Default] EasyNet::onDisconnected server 100 [2024-01-31 03:08:14.206073] [Default] EasyNet::onConnected server 100 to port 13306 [2024-01-31 03:08:14.206073] [Default] Server 2 : update server 100 's nextIndex(old:1708,new:1708) when onConnect [2024-01-31 03:08:21.655427] [Default] EasyNet::onDisconnected server 1 [2024-01-31 03:08:22.006885] [Default] EasyNet::onConnected server 1 to port 13306 [2024-01-31 03:08:22.006885] [Default] Server 2 : update server 1 's nextIndex(old:1708,new:1708) when onConnect 2024-01-31T03:08:26.365569Z 6 [System] [MY-025005] [Server] Consensus apply thread stop, opt_consensus_leader_stop_apply: false, seconds_behind_master: 111, opt_consensus_leader_stop_apply_time: 0. 2024-01-31T03:08:26.365640Z 6 [System] [MY-025006] [Server] Consensus apply thread catchup commit index, consensus index: 1697, current term: 4, stop term: 5. [2024-01-31 03:08:35.863384] [Default] EasyNet::onDisconnected server 100 [2024-01-31 03:08:44.165292] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:44.165292] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:44.365732] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:44.365732] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:44.565145] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:44.565145] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:44.765769] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:44.765769] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:44.965409] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:44.965409] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:45.165961] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:45.165961] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:45.365438] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:45.365438] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:45.565922] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:45.565922] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:45.765455] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:45.765455] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:45.966121] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:45.966121] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:46.165708] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:46.165708] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:46.365206] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:46.365206] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:46.565741] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:46.565741] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:46.765250] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:46.765250] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:47.055447] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:47.055447] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:47.165654] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:47.165654] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:47.365219] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:47.365219] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:47.565649] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:47.565649] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:47.766016] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:47.766016] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:47.965529] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:47.965529] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:48.165179] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:48.165179] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. 2024-01-31T03:08:48.456100Z 14 [System] [MY-013172] [Server] Received SHUTDOWN from user root. Shutting down mysqld (Version: 8.0.30). [2024-01-31 03:08:48.365674] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:48.365674] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:48.566098] [ERROR] failed to reconnect mysql-hkampm-mysql-2.mysql-hkampm-mysql-headless:13306 by invoking easy_connection_connect, error code: -1 [2024-01-31 03:08:48.566098] [ERROR] failed to connect server 100, now this server is disconnected, try to reconnect. [2024-01-31 03:08:48.756417] [ERROR] ThreadTimerService is stopped or destroied, can not stop the timer! [2024-01-31 03:08:48.756448] [ERROR] ThreadTimerService is stopped or destroied, can not stop the timer! 2024-01-31T03:08:49.956479Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.30) WeSQL Server - GPL, Release 5, Revision e77d836. / # 



**Expected behavior**
apecloud-mysql cluster hscale 1 to 3 success

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.
github-actions[bot] commented 7 months ago

This issue has been marked as stale because it has been open for 30 days with no activity

JashBook commented 3 months ago

kubeblocks 0.9 test pass