[BUG]Status kbcli cluster instance is inconsistent with pod status

The status of pod is ContainerCreating while the status of kbcli cluster command is always Running

Steps:

Create mysql cluster and kbcli addon enable fault-chaos-mesh


kbcli cluster describe mycluster
Name: mycluster  Created Time: Jul 11,2023 14:37 UTC+0800
NAMESPACE   CLUSTER-DEFINITION   VERSION           STATUS    TERMINATION-POLICY
default     apecloud-mysql       ac-mysql-8.0.30   Running   WipeOut

Endpoints: COMPONENT MODE INTERNAL EXTERNAL mysql ReadWrite mycluster-mysql.default.svc.cluster.local:3306

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql mycluster-mysql-0 leader Running us-central1-c gke-yjtest-default-pool-50025fd7-vtwk/10.128.0.29 Jul 11,2023 15:49 UTC+0800 mysql mycluster-mysql-1 follower Running us-central1-c gke-yjtest-default-pool-50025fd7-5090/10.128.0.26 Jul 11,2023 15:47 UTC+0800 mysql mycluster-mysql-2 follower Running us-central1-c gke-yjtest-default-pool-50025fd7-zb67/10.128.0.30 Jul 11,2023 15:45 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql false 1 / 1 1Gi / 1Gi data:20Gi standard-rwo

Images: COMPONENT TYPE IMAGE mysql mysql registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha9.20230606.gf80d546.9

Data Protection: AUTO-BACKUP BACKUP-SCHEDULE TYPE BACKUP-TTL LAST-SCHEDULE RECOVERABLE-TIME

Show cluster events: kbcli cluster list-events -n default mycluster

2. inject stop node fault to leader node

kbcli fault node stop gke-yjtest-default-pool-50025fd7-vtwk -c=gcp --region=us-central1-c --project=kubeblocks --duration=2m Secret cloud-key-secret-gcp exists under default namespace. GCPChaos node-chaos-cn65s created

k get node NAME STATUS ROLES AGE VERSION gke-yjtest-default-pool-50025fd7-5090 Ready 90m v1.26.5-gke.1200 gke-yjtest-default-pool-50025fd7-vtwk NotReady 44m v1.26.5-gke.1200 gke-yjtest-default-pool-50025fd7-zb67 Ready 16m v1.26.5-gke.1200

3. Status of pod is inconsistent when the node is back to ready that the status of kbcli cluster instance is always Running

k get node NAME STATUS ROLES AGE VERSION gke-yjtest-default-pool-50025fd7-5090 Ready 92m v1.26.5-gke.1200 gke-yjtest-default-pool-50025fd7-vtwk Ready 35s v1.26.5-gke.1200 gke-yjtest-default-pool-50025fd7-zb67 Ready 17m v1.26.5-gke.1200

k get pod NAME READY STATUS RESTARTS AGE mycluster-mysql-0 0/5 ContainerCreating 5 154m mycluster-mysql-1 5/5 Running 0 156m mycluster-mysql-2 5/5 Running 0 158m

kbcli cluster describe mycluster Name: mycluster Created Time: Jul 11,2023 14:37 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY default apecloud-mysql ac-mysql-8.0.30 Abnormal WipeOut

Endpoints: COMPONENT MODE INTERNAL EXTERNAL mysql ReadWrite mycluster-mysql.default.svc.cluster.local:3306

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql mycluster-mysql-0 leader Running us-central1-c gke-yjtest-default-pool-50025fd7-vtwk/10.128.0.31 Jul 11,2023 15:49 UTC+0800 mysql mycluster-mysql-1 leader Running us-central1-c gke-yjtest-default-pool-50025fd7-5090/10.128.0.26 Jul 11,2023 15:47 UTC+0800 mysql mycluster-mysql-2 follower Running us-central1-c gke-yjtest-default-pool-50025fd7-zb67/10.128.0.30 Jul 11,2023 15:45 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql false 1 / 1 1Gi / 1Gi data:20Gi standard-rwo

Images: COMPONENT TYPE IMAGE mysql mysql registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha9.20230606.gf80d546.9

Data Protection: AUTO-BACKUP BACKUP-SCHEDULE TYPE BACKUP-TTL LAST-SCHEDULE RECOVERABLE-TIME

Show cluster events: kbcli cluster list-events -n default mycluster

kbcli cluster list-instances mycluster NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mycluster-mysql-0 default mycluster mysql Running leader ReadWrite us-central1-c 1 / 1 1Gi / 1Gi data:20Gi gke-yjtest-default-pool-50025fd7-vtwk/10.128.0.31 Jul 11,2023 15:49 UTC+0800 mycluster-mysql-1 default mycluster mysql Running leader ReadWrite us-central1-c 1 / 1 1Gi / 1Gi data:20Gi gke-yjtest-default-pool-50025fd7-5090/10.128.0.26 Jul 11,2023 15:47 UTC+0800 mycluster-mysql-2 default mycluster mysql Running follower Readonly us-central1-c 1 / 1 1Gi / 1Gi data:20Gi gke-yjtest-default-pool-50025fd7-zb67/10.128.0.30 Jul 11,2023 15:45 UTC+0800

kubectl get cluster mycluster -o yaml apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2023-07-11T10:21:47.195015664Z" creationTimestamp: "2023-07-11T06:37:13Z" finalizers:

cluster.kubeblocks.io/finalizer generation: 2 labels: clusterdefinition.kubeblocks.io/name: apecloud-mysql clusterversion.kubeblocks.io/name: ac-mysql-8.0.30 name: mycluster namespace: default resourceVersion: "1183486" uid: 75bf5c8a-17b2-40a1-9a92-dfe850a12a39 spec: affinity: nodeLabels: {} podAntiAffinity: Preferred tenancy: SharedNode topologyKeys: [] clusterDefinitionRef: apecloud-mysql clusterVersionRef: ac-mysql-8.0.30 componentSpecs:
componentDefRef: mysql monitor: true name: mysql noCreatePDB: false replicas: 3 resources: limits: cpu: "1" memory: 1Gi requests: cpu: "1" memory: 1Gi serviceAccountName: kb-sa-mycluster volumeClaimTemplates:
- name: data spec: accessModes:
  - ReadWriteOnce resources: requests: storage: 20Gi terminationPolicy: WipeOut tolerations: [] status: clusterDefGeneration: 2 components: mysql: consensusSetStatus: followers:
  - accessMode: Readonly name: follower pod: mycluster-mysql-2 leader: accessMode: ReadWrite name: leader pod: mycluster-mysql-1 phase: Abnormal podsReady: false conditions:
lastTransitionTime: "2023-07-11T06:37:26Z" message: found uncleaned resources, requires manual deletion, check with kubectl -n default get pvc,secret,cm -l app.kubernetes.io/instance=mycluster reason: UncleanedResources status: "False" type: HaltRecovery
lastTransitionTime: "2023-07-11T06:37:26Z" message: 'The operator has started the provisioning of Cluster: mycluster' observedGeneration: 2 reason: PreCheckSucceed status: "True" type: ProvisioningStarted
lastTransitionTime: "2023-07-11T07:49:24Z" message: Successfully applied for resources observedGeneration: 2 reason: ApplyResourcesSucceed status: "True" type: ApplyResources
lastTransitionTime: "2023-07-11T10:21:47Z" message: 'pods are not ready in Components: [mysql], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady
lastTransitionTime: "2023-07-11T10:21:47Z" message: 'pods are unavailable in Components: [mysql], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 2 phase: Abnormal

k logs mycluster-mysql-0 Defaulted container "mysql" out of: mysql, metrics, vttablet, kb-checkrole, config-manager Error from server (BadRequest): container "mysql" in pod "mycluster-mysql-0" is terminated

k describe pod mycluster-mysql-0 Name: mycluster-mysql-0 Namespace: default Priority: 0 Node: gke-yjtest-default-pool-50025fd7-vtwk/10.128.0.31 Start Time: Tue, 11 Jul 2023 15:49:24 +0800 Labels: app.kubernetes.io/component=mysql app.kubernetes.io/instance=mycluster app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=apecloud-mysql app.kubernetes.io/version=ac-mysql-8.0.30 apps.kubeblocks.io/component-name=mysql apps.kubeblocks.io/workload-type=Consensus controller-revision-hash=mycluster-mysql-dcffcf98f cs.apps.kubeblocks.io/access-mode=ReadWrite kubeblocks.io/role=leader statefulset.kubernetes.io/pod-name=mycluster-mysql-0 Annotations: apps.kubeblocks.io/component-replicas: 3 cs.apps.kubeblocks.io/leader: mycluster-mysql-1 Status: Running IP: IPs: Controlled By: StatefulSet/mycluster-mysql Containers: mysql: Container ID: Image: registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha9.20230606.gf80d546.9 Image ID: Ports: 3306/TCP, 13306/TCP Host Ports: 0/TCP, 0/TCP Command: /scripts/setup.sh State: Waiting Reason: ContainerCreating Last State: Terminated Reason: ContainerStatusUnknown Message: The container could not be located when the pod was deleted. The container used to be Running Exit Code: 137 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Mon, 01 Jan 0001 00:00:00 +0000 Ready: False Restart Count: 1 Limits: cpu: 1 memory: 1Gi Requests: cpu: 1 memory: 1Gi Environment Variables from: mycluster-mysql-env ConfigMap Optional: false Environment: KB_POD_NAME: mycluster-mysql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: mycluster KB_COMP_NAME: mysql KB_CLUSTER_COMP_NAME: mycluster-mysql KB_CLUSTER_UID_POSTFIX_8: 50a12a39 KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc MYSQL_ROOT_HOST: % MYSQL_ROOT_USER: <set to the key 'username' in secret 'mycluster-conn-credential'> Optional: false MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'mycluster-conn-credential'> Optional: false MYSQL_DATABASE: mydb MYSQL_USER: u1 MYSQL_PASSWORD: u1 CLUSTER_ID: 1 CLUSTER_START_INDEX: 1 REPLICATION_USER: replicator REPLICATION_PASSWORD: MYSQL_TEMPLATE_CONFIG: MYSQL_CUSTOM_CONFIG: MYSQL_DYNAMIC_CONFIG: KB_EMBEDDED_WESQL: 1 Mounts: /data/mysql from data (rw) /etc/annotations from annotations (rw) /opt/mysql from mysql-config (rw) /scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcspw (ro) metrics: Container ID: Image: registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1 Image ID: Port: 9104/TCP Host Port: 0/TCP Command: /bin/agamotto --config=/opt/agamotto/agamotto-config.yaml State: Waiting Reason: ContainerCreating Last State: Terminated Reason: ContainerStatusUnknown Message: The container could not be located when the pod was deleted. The container used to be Running Exit Code: 137 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Mon, 01 Jan 0001 00:00:00 +0000 Ready: False Restart Count: 1 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: mycluster-mysql-env ConfigMap Optional: false Environment: KB_POD_NAME: mycluster-mysql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: mycluster KB_COMP_NAME: mysql KB_CLUSTER_COMP_NAME: mycluster-mysql KB_CLUSTER_UID_POSTFIX_8: 50a12a39 KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc DB_TYPE: MySQL ENDPOINT: localhost:3306 MYSQL_USER: <set to the key 'username' in secret 'mycluster-conn-credential'> Optional: false MYSQL_PASSWORD: <set to the key 'password' in secret 'mycluster-conn-credential'> Optional: false Mounts: /data/mysql from data (rw) /opt/agamotto from agamotto-configuration (rw) /var/log/kubeblocks from log-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcspw (ro) vttablet: Container ID: Image: registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:latest Image ID: Ports: 15100/TCP, 16100/TCP, 40000/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP Command: /scripts/vttablet.sh State: Waiting Reason: ContainerCreating Last State: Terminated Reason: ContainerStatusUnknown Message: The container could not be located when the pod was deleted. The container used to be Running Exit Code: 137 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Mon, 01 Jan 0001 00:00:00 +0000 Ready: False Restart Count: 1 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: mycluster-mysql-env ConfigMap Optional: false Environment: KB_POD_NAME: mycluster-mysql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: mycluster KB_COMP_NAME: mysql KB_CLUSTER_COMP_NAME: mycluster-mysql KB_CLUSTER_UID_POSTFIX_8: 50a12a39 KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc CELL: zone1 ETCD_SERVER: $(KB_CLUSTER_NAME)-etcd-headless ETCD_PORT: 2379 TOPOLOGY_FLAGS: --topo_implementation etcd2 --topo_global_server_address $(ETCD_SERVER):$(ETCD_PORT) --topo_global_root /vitess/global VTTABLET_PORT: 15100 VTTABLET_GRPC_PORT: 16100 VTCTLD_HOST: $(KB_CLUSTER_NAME)-vtctld-headless VTCTLD_WEB_PORT: 15000 MYSQL_ROOT_USER: <set to the key 'username' in secret 'mycluster-conn-credential'> Optional: false MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'mycluster-conn-credential'> Optional: false Mounts: /conf from mysql-scale-config (rw) /scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcspw (ro) kb-checkrole: Container ID: Image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.34 Image ID: Ports: 3501/TCP, 50001/TCP Host Ports: 0/TCP, 0/TCP Command: probe --app-id batch-sdk --dapr-http-port 3501 --dapr-grpc-port 50001 --log-level info --config /config/probe/config.yaml --components-path /config/probe/components State: Waiting Reason: ContainerCreating Last State: Terminated Reason: ContainerStatusUnknown Message: The container could not be located when the pod was deleted. The container used to be Running Exit Code: 137 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Mon, 01 Jan 0001 00:00:00 +0000 Ready: False Restart Count: 1 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Readiness: http-get http://:3501/v1.0/bindings/mysql%3Foperation=checkRole delay=0s timeout=1s period=1s #success=1 #failure=2 Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: mycluster-mysql-env ConfigMap Optional: false Environment: KB_POD_NAME: mycluster-mysql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: mycluster KB_COMP_NAME: mysql KB_CLUSTER_COMP_NAME: mycluster-mysql KB_CLUSTER_UID_POSTFIX_8: 50a12a39 KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc KB_SERVICE_USER: <set to the key 'username' in secret 'mycluster-conn-credential'> Optional: false KB_SERVICE_PASSWORD: <set to the key 'password' in secret 'mycluster-conn-credential'> Optional: false KB_SERVICE_PORT: 3306 KB_SERVICE_ROLES: {"follower":"Readonly","leader":"ReadWrite","learner":"Readonly"} KB_SERVICE_CHARACTER_TYPE: mysql Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcspw (ro) config-manager: Container ID: Image: registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.34 Image ID: Port: Host Port: Command: env Args: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH) /bin/reloader --log-level info --volume-dir /conf --operator-update-enable --tcp 9901 --config /opt/config-manager/config-manager.yaml State: Waiting Reason: ContainerCreating Last State: Terminated Reason: ContainerStatusUnknown Message: The container could not be located when the pod was deleted. The container used to be Running Exit Code: 137 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Mon, 01 Jan 0001 00:00:00 +0000 Ready: False Restart Count: 1 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: mycluster-mysql-env ConfigMap Optional: false Environment: KB_POD_NAME: mycluster-mysql-0 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_CLUSTER_NAME: mycluster KB_COMP_NAME: mysql KB_CLUSTER_COMP_NAME: mycluster-mysql KB_CLUSTER_UID_POSTFIX_8: 50a12a39 KB_POD_FQDN: $(KB_POD_NAME).$(KB_CLUSTER_COMP_NAME)-headless.$(KB_NAMESPACE).svc CONFIG_MANAGER_POD_IP: (v1:status.podIP) DB_TYPE: mysql MYSQL_USER: <set to the key 'username' in secret 'mycluster-conn-credential'> Optional: false MYSQL_PASSWORD: <set to the key 'password' in secret 'mycluster-conn-credential'> Optional: false DATA_SOURCE_NAME: $(MYSQL_USER):$(MYSQL_PASSWORD)@(localhost:3306)/ TOOLS_PATH: /opt/kb-tools/reload/mysql-consensusset-config:/opt/config-manager Mounts: /conf from mysql-scale-config (rw) /opt/config-manager from config-manager-config (rw) /opt/kb-tools/reload/mysql-consensusset-config from cm-script-mysql-consensusset-config (rw) /opt/mysql from mysql-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-fcspw (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-mycluster-mysql-0 ReadOnly: false log-data: Type: HostPath (bare host directory volume) Path: /var/log/kubeblocks HostPathType: DirectoryOrCreate annotations: Type: DownwardAPI (a volume populated by information about the pod) Items: metadata.annotations['cs.apps.kubeblocks.io/leader'] -> leader metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas agamotto-configuration: Type: ConfigMap (a volume populated by a ConfigMap) Name: mycluster-mysql-agamotto-configuration Optional: false scripts: Type: ConfigMap (a volume populated by a ConfigMap) Name: mycluster-mysql-apecloud-mysql-scripts Optional: false mysql-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: mycluster-mysql-mysql-consensusset-config Optional: false mysql-scale-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: mycluster-mysql-mysql-scale-config Optional: false cm-script-mysql-consensusset-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-mysql-reload-script-mycluster Optional: false config-manager-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-mycluster-mysql-config-manager-config Optional: false kube-api-access-fcspw: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: kb-data=true:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message

Normal checkRole 59m Warning Unhealthy 51m Warning FailedMount Warning FailedMount 46m Normal Pulling 46m Normal Created 36m Normal Started 36m Normal Pulling 36m Normal Pulled 36m Normal Pulled 36m Normal Created 36m Normal Pulling 36m Normal Started 36m Normal Pulled 35m Normal Created 35m Normal Started 34m Normal Pulling 34m Normal Pulled 34m Normal Created 34m Normal Started 34m Normal Pulled 34m Normal Created 34m Normal checkRole 34m Normal checkRole 21m Warning Unhealthy 5m11s Warning NodeNotReady Warning NetworkNotReady 2m48s Warning FailedMount Warning FailedMount Warning FailedMount Warning FailedMount Warning FailedMount Warning FailedMount Warning FailedMount Warning FailedMount sqlchannel {"event":"Success","operation":"checkRole","originalRole":"Follower","role":"Leader"} kubelet Readiness probe failed: Get "http://10.104.2.7:3501/v1.0/bindings/mysql?operation=checkRole": dial tcp 10.104.2.7:3501: connect: connection refused 47m (x8 over 48m) kubelet MountVolume.NodeAffinity check failed for volume "pvc-f5fcacf2-56ea-48c5-9963-32d149029ddd" : no matching NodeSelectorTerms kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[log-data config-manager-config mysql-scale-config annotations kube-api-access-fcspw agamotto-configuration cm-script-mysql-consensusset-config data mysql-config scripts]: timed out waiting for the condition kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha9.20230606.gf80d546.9" kubelet Created container mysql kubelet Started container mysql kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1" kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.alpha9.20230606.gf80d546.9" in 9m49.275940754s (9m49.275949118s including waiting) kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1" in 24.631034884s (24.631046851s including waiting) kubelet Created container metrics kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:latest" kubelet Started container metrics kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/apecloud-mysql-scale:latest" in 1m17.915772496s (1m17.915787768s including waiting) kubelet Created container vttablet kubelet Started container vttablet kubelet Pulling image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.34" kubelet Successfully pulled image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.34" in 28.096796489s (28.096809048s including waiting) kubelet Created container kb-checkrole kubelet Started container kb-checkrole kubelet Container image "registry.cn-hangzhou.aliyuncs.com/apecloud/kubeblocks-tools:0.6.0-alpha.34" already present on machine kubelet Created container config-manager sqlchannel {"event":"Success","operation":"checkRole","originalRole":"","role":"Follower"} sqlchannel {"event":"Success","operation":"checkRole","originalRole":"Follower","role":"Leader"} kubelet Readiness probe failed: Get "http://10.104.2.7:3501/v1.0/bindings/mysql?operation=checkRole": dial tcp 10.104.2.7:3501: connect: connection refused 4m33s (x3 over 72m) node-controller Node is not ready kubelet network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "agamotto-configuration" : object "default"/"mycluster-mysql-agamotto-configuration" not registered 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "mysql-scale-config" : object "default"/"mycluster-mysql-mysql-scale-config" not registered 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "scripts" : object "default"/"mycluster-mysql-apecloud-mysql-scripts" not registered 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "kube-api-access-fcspw" : object "default"/"kube-root-ca.crt" not registered 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "mysql-config" : object "default"/"mycluster-mysql-mysql-consensusset-config" not registered 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "cm-script-mysql-consensusset-config" : object "default"/"sidecar-mysql-reload-script-mycluster" not registered 2m46s (x3 over 2m48s) kubelet MountVolume.SetUp failed for volume "config-manager-config" : object "default"/"sidecar-mycluster-mysql-config-manager-config" not registered 2m35s (x3 over 2m38s) kubelet MountVolume.MountDevice failed for volume "pvc-f5fcacf2-56ea-48c5-9963-32d149029ddd" : rpc error: code = Internal desc = Error when getting device path: rpc error: code = Internal desc = error verifying GCE PD ("pvc-f5fcacf2-56ea-48c5-9963-32d149029ddd") is attached: failed to find and re-link disk pvc-f5fcacf2-56ea-48c5-9963-32d149029ddd with udevadm after retrying for 3s: for disk pvc-f5fcacf2-56ea-48c5-9963-32d149029ddd failed to trigger udevadm fix of non existent device path: udevadm --trigger requested to fix disk pvc-f5fcacf2-56ea-48c5-9963-32d149029ddd but no such disk was found in device path map[/dev/sda:persistent-disk-0 /dev/sda1:persistent-disk-0 /dev/sda10:persistent-disk-0 /dev/sda11:persistent-disk-0 /dev/sda12:persistent-disk-0 /dev/sda2:persistent-disk-0 /dev/sda3:persistent-disk-0 /dev/sda4:persistent-disk-0 /dev/sda5:persistent-disk-0 /dev/sda6:persistent-disk-0 /dev/sda7:persistent-disk-0 /dev/sda8:persistent-disk-0 /dev/sda9:persistent-disk-0]

apecloud / kubeblocks

[BUG]Status kbcli cluster instance is inconsistent with pod status #4217