search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.05k stars 167 forks source link

[BUG] restart a component with evicted pod failed #8027

Open nayutah opened 2 weeks ago

nayutah commented 2 weeks ago

Describe the bug A mysql cluster has a mysql component with two replicas, and one replica is evicted due to disk pressure of node ephemeral-storage. When the disk pressure recovers to normal (< 85%) state, the evicted Pod is still in ContainerStatusUnknown state, and we start a 'restart' ops to restart the mysql cluster, then the ops is stuck and two replicas are in ContainerStatusUnknown state.

To Reproduce Steps to reproduce the behavior:

  1. Create a mysql primary-secondary cluster
  2. Make the system disk with a disk pressure level more than 85%
  3. One of the mysql replica is evicted, and in state ContainerStatusUnknown
  4. Restart the mysql cluster
  5. Two replicas are both in ContainerStatusUnknown state

Expected behavior 1, Reject to restart for a failed or abnormal component 2, Or permit restarting the failed/abnormal component, and goes well

Screenshots

Desktop (please complete the following information):

Additional context Add any other context about the problem here.

nayutah commented 2 weeks ago

Ops object: kubectl describe ops daisy-776f7959cb-restart-zvtt4 -n kubeblocks-cloud-ns Name: daisy-776f7959cb-restart-zvtt4 Namespace: kubeblocks-cloud-ns Labels: app.kubernetes.io/instance=daisy-776f7959cb cloud.kubeblocks.io/cluster=daisy69 cloud.kubeblocks.io/organization=dullboy ops.kubeblocks.io/ops-type=Restart Annotations: API Version: apps.kubeblocks.io/v1alpha1 Kind: OpsRequest Metadata: Creation Timestamp: 2024-08-27T06:20:55Z Finalizers: opsrequest.kubeblocks.io/finalizer Generate Name: daisy-776f7959cb-restart- Generation: 1 Managed Fields: API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:generateName: f:labels: .: f:cloud.kubeblocks.io/cluster: f:cloud.kubeblocks.io/organization: f:spec: .: f:clusterName: f:enqueueOnForce: f:preConditionDeadlineSeconds: f:restart: .: k:{"componentName":"mysql"}: .: f:componentName: f:type: Manager: apiserver Operation: Update Time: 2024-08-27T06:20:55Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: v:"opsrequest.kubeblocks.io/finalizer": f:labels: f:app.kubernetes.io/instance: f:ops.kubeblocks.io/ops-type: f:ownerReferences: .: k:{"uid":"6969684f-e816-4643-bb10-6fb49b8312c5"}: Manager: kubeblocks Operation: Update Time: 2024-08-27T06:20:55Z API Version: apps.kubeblocks.io/v1alpha1 Fields Type: FieldsV1 fieldsV1: f:status: .: f:clusterGeneration: f:components: .: f:mysql: .: f:phase: f:progressDetails: f:conditions: .: k:{"type":"Restarting"}: .: f:lastTransitionTime: f:message: f:reason: f:status: f:type: k:{"type":"Validated"}: .: f:lastTransitionTime: f:message: f:reason: f:status: f:type: k:{"type":"WaitForProgressing"}: .: f:lastTransitionTime: f:message: f:reason: f:status: f:type: f:phase: f:progress: f:startTimestamp: Manager: kubeblocks Operation: Update Subresource: status Time: 2024-08-27T06:20:56Z Owner References: API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Name: daisy-776f7959cb UID: 6969684f-e816-4643-bb10-6fb49b8312c5 Resource Version: 383869986 UID: ed38082c-cb58-41a9-9478-cdb3a27141a8 Spec: Cluster Name: daisy-776f7959cb Enqueue On Force: false Pre Condition Deadline Seconds: 0 Restart: Component Name: mysql Type: Restart Status: Cluster Generation: 1 Components: Mysql: Phase: Failed Progress Details: Object Key: Pod/daisy-776f7959cb-mysql-1 Status: Pending Object Key: Pod/daisy-776f7959cb-mysql-0 Status: Pending Conditions: Last Transition Time: 2024-08-27T06:20:55Z Message: wait for the controller to process the OpsRequest: daisy-776f7959cb-restart-zvtt4 in Cluster: daisy-776f7959cb Reason: WaitForProgressing Status: True Type: WaitForProgressing Last Transition Time: 2024-08-27T06:20:55Z Message: OpsRequest: daisy-776f7959cb-restart-zvtt4 is validated Reason: ValidateOpsRequestPassed Status: True Type: Validated Last Transition Time: 2024-08-27T06:20:55Z Message: Start to restart database in Cluster: daisy-776f7959cb Reason: RestartStarted Status: True Type: Restarting Phase: Running Progress: 0/2 Start Timestamp: 2024-08-27T06:20:55Z Events: Type Reason Age From Message

Normal WaitForProgressing 12m ops-request-controller wait for the controller to process the OpsRequest: daisy-776f7959cb-restart-zvtt4 in Cluster: daisy-776f7959cb Normal ValidateOpsRequestPassed 12m (x2 over 12m) ops-request-controller OpsRequest: daisy-776f7959cb-restart-zvtt4 is validated Normal RestartStarted 12m (x2 over 12m) ops-request-controller Start to restart database in Cluster: daisy-776f7959cb

nayutah commented 2 weeks ago

slc@slcmac aws % kg pods -A | grep daisy kubeblocks-cloud-ns 6969684f-daisy-776f7959cb-mysq-archive-binlog-0 1/2 CrashLoopBackOff 6 (47s ago) 176m kubeblocks-cloud-ns daisy-776f7959cb-mysql-0 0/4 ContainerStatusUnknown 3 3h13m kubeblocks-cloud-ns daisy-776f7959cb-mysql-1 0/4 ContainerStatusUnknown 3 3h13m

kubectl describe pod daisy-776f7959cb-mysql-1 Name: daisy-776f7959cb-mysql-1 Namespace: kubeblocks-cloud-ns Priority: 0 Node: cn-zhangjiakou.192.168.0.202/192.168.0.202 Start Time: Tue, 27 Aug 2024 11:20:25 +0800 Labels: app.kubernetes.io/component=mysql app.kubernetes.io/instance=daisy-776f7959cb app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=mysql app.kubernetes.io/version= apps.kubeblocks.io/cluster-uid=6969684f-e816-4643-bb10-6fb49b8312c5 apps.kubeblocks.io/component-name=mysql apps.kubeblocks.io/pod-name=daisy-776f7959cb-mysql-1 cloud.kubeblocks.io/cluster=daisy69 cloud.kubeblocks.io/organization=dullboy clusterdefinition.kubeblocks.io/name=mysql clusterversion.kubeblocks.io/name=mysql-8.0.33 controller-revision-hash=67c454645f kubeblocks.io/role=secondary workloads.kubeblocks.io/access-mode=Readonly workloads.kubeblocks.io/instance=daisy-776f7959cb-mysql workloads.kubeblocks.io/managed-by=InstanceSet Annotations: apps.kubeblocks.io/component-replicas: 2 apps.kubeblocks.io/last-role-snapshot-version: 1724729870754711 k8s.aliyun.com/pod-ips: 192.168.0.23 Status: Failed Reason: Evicted Message: The node was low on resource: ephemeral-storage. Threshold quantity: 7898263456, available: 7194828Ki. Container config-manager was using 72Ki, request is 0, has larger consumption of ephemeral-storage. Container lorry was using 48Ki, request is 0, has larger consumption of ephemeral-storage. Container mysql was using 496Ki, request is 0, has larger consumption of ephemeral-storage. Container metrics was using 64Ki, request is 0, has larger consumption of ephemeral-storage. IP: 192.168.0.23 IPs: IP: 192.168.0.23 Controlled By: InstanceSet/daisy-776f7959cb-mysql Init Containers: init-data: Container ID: containerd://681633fbece95d2c57b9b035d42c158d29dcc990ee6579ede61f0c2ebed46abc Image: apecloud-registry-vpc.cn-zhangjiakou.cr.aliyuncs.com/apecloud/mysql_audit_log:8.0.33 Image ID: apecloud-registry-vpc.cn-zhangjiakou.cr.aliyuncs.com/apecloud/mysql_audit_log@sha256:46c02c388384e2da0a43d520c26bb6d52b9e1ca92009db25acee58f9c0df8ccf Port: Host Port: Command: bash -c mkdir -p /var/lib/mysql/{log,binlog,auditlog} cp /usr/lib/mysql/plugin/ /var/lib/mysql/plugin -r

State:          Terminated
  Reason:       Completed
  Exit Code:    0
  Started:      Tue, 27 Aug 2024 11:20:26 +0800
  Finished:     Tue, 27 Aug 2024 11:20:26 +0800
Ready:          True
Restart Count:  0
Limits:
  cpu:     0
  memory:  0
Requests:
  cpu:     0
  memory:  0
Environment Variables from:
  daisy-776f7959cb-mysql-env  ConfigMap  Optional: false
Environment:
  KB_POD_NAME:   daisy-776f7959cb-mysql-1 (v1:metadata.name)
  KB_POD_UID:     (v1:metadata.uid)
  KB_NAMESPACE:  kubeblocks-cloud-ns (v1:metadata.namespace)
  KB_SA_NAME:     (v1:spec.serviceAccountName)
  KB_NODENAME:    (v1:spec.nodeName)
  KB_HOST_IP:     (v1:status.hostIP)
  KB_POD_IP:      (v1:status.podIP)
  KB_POD_IPS:     (v1:status.podIPs)
  KB_HOSTIP:      (v1:status.hostIP)
  KB_PODIP:       (v1:status.podIP)
  KB_PODIPS:      (v1:status.podIPs)
  KB_POD_FQDN:   $(KB_POD_NAME).daisy-776f7959cb-mysql-headless.$(KB_NAMESPACE).svc
Mounts:
  /var/lib/mysql from data (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p2gjk (ro)

init-syncer: Container ID: containerd://bcd27823095f2f273d548feca7c725321b2f8db6596582986a6a35a8a1140da1 Image: apecloud-registry-vpc.cn-zhangjiakou.cr.aliyuncs.com/apecloud/syncer:0.1.2 Image ID: apecloud-registry-vpc.cn-zhangjiakou.cr.aliyuncs.com/apecloud/syncer@sha256:1ef2cbdc120acbebebef55236fb36f78fd95259db3c4f68c30071f2a490e4994 Port: Host Port: Command: cp -r /bin/syncer /config /kubeblocks/ State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 27 Aug 2024 11:20:26 +0800 Finished: Tue, 27 Aug 2024 11:20:26 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: daisy-776f7959cb-mysql-env ConfigMap Optional: false Environment: KB_POD_NAME: daisy-776f7959cb-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: kubeblocks-cloud-ns (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).daisy-776f7959cb-mysql-headless.$(KB_NAMESPACE).svc Mounts: /kubeblocks from kubeblocks (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p2gjk (ro) init-xtrabackup: Container ID: containerd://d65e47061a270e3090c95ddca1147394de98a6884358fa3448193561b7b1157a Image: apecloud-registry-vpc.cn-zhangjiakou.cr.aliyuncs.com/apecloud/syncer:mysql Image ID: apecloud-registry-vpc.cn-zhangjiakou.cr.aliyuncs.com/apecloud/syncer@sha256:a97171ee1cea23efbb5ce146b13fcf7478f309bf0a25cd11bbfcfd5e718ee251 Port: Host Port: Command: cp -r /xtrabackup-8.0 /kubeblocks/xtrabackup State: Terminated Reason: Completed Exit Code: 0 Started: Tue, 27 Aug 2024 11:20:27 +0800 Finished: Tue, 27 Aug 2024 11:20:37 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: daisy-776f7959cb-mysql-env ConfigMap Optional: false Environment: KB_POD_NAME: daisy-776f7959cb-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: kubeblocks-cloud-ns (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).daisy-776f7959cb-mysql-headless.$(KB_NAMESPACE).svc Mounts: /kubeblocks from kubeblocks (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p2gjk (ro) Containers: mysql: Container ID: Image: apecloud-registry-vpc.cn-zhangjiakou.cr.aliyuncs.com/apecloud/mysql:8.0.33 Image ID: Ports: 3306/TCP, 3601/TCP Host Ports: 0/TCP, 0/TCP Command: syncer --config-path /kubeblocks/config/components --port 3601

  bash
  -c
  mv /var/lib/mysql/plugin/audit_log.so /usr/lib64/mysql/plugin/
  rm -rf /var/lib/mysql/plugin
  chown -R mysql:root /var/lib/mysql
  skip_replica_start="OFF"
  if [ -f /var/lib/mysql/data/.restore_new_cluster ]; then
    skip_replica_start="ON"
  fi
  docker-entrypoint.sh mysqld --server-id $(( ${KB_POD_NAME##*-} + 1)) \
  --plugin-load-add=rpl_semi_sync_source=semisync_source.so \
  --plugin-load-add=rpl_semi_sync_replica=semisync_replica.so \
  --plugin-load-add=audit_log=audit_log.so \
  --log-bin=/var/lib/mysql/binlog/$(KB_POD_NAME)-bin \
  --skip-replica-start=$skip_replica_start

State:          Terminated
  Reason:       ContainerStatusUnknown
  Message:      The container could not be located when the pod was terminated
  Exit Code:    137
  Started:      Mon, 01 Jan 0001 00:00:00 +0000
  Finished:     Mon, 01 Jan 0001 00:00:00 +0000
Last State:     Terminated
  Reason:       ContainerStatusUnknown
  Message:      The container could not be located when the pod was deleted.  The container used to be Running
  Exit Code:    137
  Started:      Mon, 01 Jan 0001 00:00:00 +0000
  Finished:     Mon, 01 Jan 0001 00:00:00 +0000
Ready:          False
Restart Count:  1
Limits:
  cpu:     1
  memory:  2Gi
Requests:
  cpu:     100m
  memory:  204Mi
Environment Variables from:
  daisy-776f7959cb-mysql-env      ConfigMap  Optional: false
  daisy-776f7959cb-mysql-rsm-env  ConfigMap  Optional: false
Environment:
  KB_POD_NAME:                daisy-776f7959cb-mysql-1 (v1:metadata.name)
  KB_POD_UID:                  (v1:metadata.uid)
  KB_NAMESPACE:               kubeblocks-cloud-ns (v1:metadata.namespace)
  KB_SA_NAME:                  (v1:spec.serviceAccountName)
  KB_NODENAME:                 (v1:spec.nodeName)
  KB_HOST_IP:                  (v1:status.hostIP)
  KB_POD_IP:                   (v1:status.podIP)
  KB_POD_IPS:                  (v1:status.podIPs)
  KB_HOSTIP:                   (v1:status.hostIP)
  KB_PODIP:                    (v1:status.podIP)
  KB_PODIPS:                   (v1:status.podIPs)
  KB_POD_FQDN:                $(KB_POD_NAME).daisy-776f7959cb-mysql-headless.$(KB_NAMESPACE).svc
  PATH:                       /kubeblocks/xtrabackup/bin:/kubeblocks/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  KB_SERVICE_CHARACTER_TYPE:  mysql
  MYSQL_INITDB_SKIP_TZINFO:   1
  SERVICE_PORT:               3306
  MYSQL_ROOT_HOST:            %
  MYSQL_ROOT_USER:            <set to the key 'username' in secret 'daisy-776f7959cb-conn-credential'>  Optional: false
  MYSQL_ROOT_PASSWORD:        <set to the key 'password' in secret 'daisy-776f7959cb-conn-credential'>  Optional: false
Mounts:
  /etc/mysql/conf.d from mysql-config (rw)
  /kubeblocks from kubeblocks (rw)
  /scripts from scripts (rw)
  /var/lib/mysql from data (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p2gjk (ro)

metrics: Container ID: containerd://1a491ad694b33b619cf64f75acc31a5bd6bb1506836bfa4830fb30bdf27cdc86 Image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1 Image ID: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto@sha256:01d06a16a2e96d0c5e1ac97db40353abad8ee73f34f9f7d148984e7de4ff0bd5 Port: 9104/TCP Host Port: 0/TCP Command: /scripts/agamotto.sh State: Terminated Reason: Error Exit Code: 137 Started: Tue, 27 Aug 2024 11:20:49 +0800 Finished: Tue, 27 Aug 2024 12:18:10 +0800 Ready: False Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: daisy-776f7959cb-mysql-env ConfigMap Optional: false daisy-776f7959cb-mysql-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: daisy-776f7959cb-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: kubeblocks-cloud-ns (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).daisy-776f7959cb-mysql-headless.$(KB_NAMESPACE).svc DB_TYPE: MySQL ENDPOINT: localhost:3306 MYSQL_USER: <set to the key 'username' in secret 'daisy-776f7959cb-conn-credential'> Optional: false MYSQL_PASSWORD: <set to the key 'password' in secret 'daisy-776f7959cb-conn-credential'> Optional: false Mounts: /data/mysql from data (rw) /opt/agamotto from agamotto-configuration (rw) /scripts from scripts (rw) /var/log/kubeblocks from log-data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p2gjk (ro) lorry: Container ID: Image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.9.1-beta.15 Image ID: Ports: 3501/TCP, 50001/TCP Host Ports: 0/TCP, 0/TCP Command: lorry --port 3501 --grpcport 50001 State: Terminated Reason: ContainerStatusUnknown Message: The container could not be located when the pod was terminated Exit Code: 137 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Mon, 01 Jan 0001 00:00:00 +0000 Last State: Terminated Reason: ContainerStatusUnknown Message: The container could not be located when the pod was deleted. The container used to be Running Exit Code: 137 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Mon, 01 Jan 0001 00:00:00 +0000 Ready: False Restart Count: 1 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Readiness: http-get http://:3501/v1.0/checkrole delay=0s timeout=1s period=1s #success=1 #failure=3 Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: daisy-776f7959cb-mysql-env ConfigMap Optional: false daisy-776f7959cb-mysql-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: daisy-776f7959cb-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: kubeblocks-cloud-ns (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).daisy-776f7959cb-mysql-headless.$(KB_NAMESPACE).svc KB_RSM_ROLE_PROBE_PERIOD: 1 KB_BUILTIN_HANDLER: mysql KB_SERVICE_USER: <set to the key 'username' in secret 'daisy-776f7959cb-conn-credential'> Optional: false KB_SERVICE_PASSWORD: <set to the key 'password' in secret 'daisy-776f7959cb-conn-credential'> Optional: false KB_SERVICE_PORT: 3306 KB_DATA_PATH: /var/lib/mysql KB_RSM_ACTION_SVC_LIST: null KB_RSM_ROLE_UPDATE_MECHANISM: DirectAPIServerEventUpdate KB_RSM_ROLE_PROBE_TIMEOUT: 1 KB_CLUSTER_NAME: (v1:metadata.labels['app.kubernetes.io/instance']) KB_COMP_NAME: (v1:metadata.labels['apps.kubeblocks.io/component-name']) KB_SERVICE_CHARACTER_TYPE: mysql Mounts: /var/lib/mysql from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p2gjk (ro) config-manager: Container ID: Image: apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/kubeblocks-tools:0.9.1-beta.15 Image ID: Port: 9901/TCP Host Port: 0/TCP Command: env Args: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH) /bin/reloader --log-level info --operator-update-enable --tcp 9901 --config /opt/config-manager/config-manager.yaml State: Terminated Reason: ContainerStatusUnknown Message: The container could not be located when the pod was terminated Exit Code: 137 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Mon, 01 Jan 0001 00:00:00 +0000 Last State: Terminated Reason: ContainerStatusUnknown Message: The container could not be located when the pod was deleted. The container used to be Running Exit Code: 137 Started: Mon, 01 Jan 0001 00:00:00 +0000 Finished: Mon, 01 Jan 0001 00:00:00 +0000 Ready: False Restart Count: 1 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: daisy-776f7959cb-mysql-env ConfigMap Optional: false daisy-776f7959cb-mysql-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: daisy-776f7959cb-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: kubeblocks-cloud-ns (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).daisy-776f7959cb-mysql-headless.$(KB_NAMESPACE).svc CONFIG_MANAGER_POD_IP: (v1:status.podIP) DB_TYPE: mysql MYSQL_USER: <set to the key 'username' in secret 'daisy-776f7959cb-conn-credential'> Optional: false MYSQL_PASSWORD: <set to the key 'password' in secret 'daisy-776f7959cb-conn-credential'> Optional: false DATA_SOURCE_NAME: $(MYSQL_USER):$(MYSQL_PASSWORD)@(localhost:3306)/ TOOLS_PATH: /opt/kb-tools/reload/mysql-replication-config:/opt/config-manager Mounts: /etc/mysql/conf.d from mysql-config (rw) /opt/config-manager from config-manager-config (rw) /opt/kb-tools/reload/mysql-replication-config from cm-script-mysql-replication-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-p2gjk (ro) Conditions: Type Status DisruptionTarget True Initialized True Ready False ContainersReady False PodScheduled True Volumes: agamotto-configuration: Type: ConfigMap (a volume populated by a ConfigMap) Name: daisy-776f7959cb-mysql-agamotto-configuration Optional: false mysql-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: daisy-776f7959cb-mysql-mysql-replication-config Optional: false scripts: Type: ConfigMap (a volume populated by a ConfigMap) Name: daisy-776f7959cb-mysql-mysql-scripts Optional: false cm-script-mysql-replication-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-oracle-mysql8.0-reload-script-daisy-776f7959cb Optional: false config-manager-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-daisy-776f7959cb-mysql-config-manager-config Optional: false data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-daisy-776f7959cb-mysql-1 ReadOnly: false kubeblocks: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: log-data: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: kube-api-access-p2gjk: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: cloud.kubeblocks.io/node-group:NoSchedule op=Exists kb-data=true:NoSchedule node-role.cloud.kubeblocks.io/control-plane:NoSchedule op=Exists node-role.cloud.kubeblocks.io/data-plane:NoSchedule op=Exists node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events:

nayutah commented 2 weeks ago

Operator logs: 2024-08-27T06:20:55.743Z INFO reconcile {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-restart-zvtt4", "reconcileID": "593d06d9-0452-481f-9ee1-3390a3c048d6", "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:20:55.756Z INFO reconcile {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-restart-zvtt4", "reconcileID": "2450b583-7f42-4d7c-a45d-835a2c6a3a06", "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:20:55.778Z INFO reconcile {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-restart-zvtt4", "reconcileID": "f5146722-43fc-47cc-9820-407045778b53", "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:20:55.790Z INFO reconcile {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-restart-zvtt4", "reconcileID": "5542435f-28f6-4012-b50e-c393021cd45f", "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:20:55.832Z INFO reconcile {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-restart-zvtt4", "reconcileID": "91933a62-f252-4147-b5b1-b89242f7aa5d", "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:20:55.847Z INFO reconcile {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-restart-zvtt4", "reconcileID": "141aef05-afe0-4699-a803-149a31bd0500", "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:20:55.884Z INFO reconcile {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-restart-zvtt4", "reconcileID": "561a0385-58b4-4e4c-8fd1-b732d0e7e532", "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:20:55.937Z INFO reconcile {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-restart-zvtt4", "reconcileID": "024583f9-5688-4038-bb14-d34e9c2df981", "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:20:55.958Z INFO status conditions, creating: false, available: false, its running: false, has failure: true, updating: false, config synced: true {"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"daisy-776f7959cb-mysql","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-mysql", "reconcileID": "9f77b3df-64a1-4d4b-91e9-f3f7a10ee79b", "component": {"name":"daisy-776f7959cb-mysql","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:20:55.970Z INFO cluster component phase transition: Abnormal -> Failed (component is Failed) {"controller": "cluster", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Cluster", "Cluster": {"name":"daisy-776f7959cb","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb", "reconcileID": "48c46d44-047c-4a02-ab6c-8016c6bee13f", "cluster": {"name":"daisy-776f7959cb","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:20:56.007Z INFO reconcile {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-restart-zvtt4", "reconcileID": "b4055520-b314-46f4-8d3b-cde26be1e7af", "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:20:56.020Z INFO reconcile {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-restart-zvtt4", "reconcileID": "fab9ab8e-5039-496f-9e03-a7c7ce219ff2", "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:20:56.115Z INFO status conditions, creating: false, available: false, its running: false, has failure: true, updating: false, config synced: true {"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"daisy-776f7959cb-mysql","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-mysql", "reconcileID": "a122aef3-a2eb-43a8-8b0f-cfcdf79a0f4a", "component": {"name":"daisy-776f7959cb-mysql","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:28:15.656Z INFO reconcile {"controller": "opsrequest", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "OpsRequest", "OpsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-restart-zvtt4", "reconcileID": "81a37814-ac0d-4745-a1fb-598cb5895363", "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}, "opsRequest": {"name":"daisy-776f7959cb-restart-zvtt4","namespace":"kubeblocks-cloud-ns"}} 2024-08-27T06:28:15.819Z INFO status conditions, creating: false, available: false, its running: false, has failure: true, updating: false, config synced: true {"controller": "component", "controllerGroup": "apps.kubeblocks.io", "controllerKind": "Component", "Component": {"name":"daisy-776f7959cb-mysql","namespace":"kubeblocks-cloud-ns"}, "namespace": "kubeblocks-cloud-ns", "name": "daisy-776f7959cb-mysql", "reconcileID": "42f15a51-7392-4733-af9f-d50fb039f448", "component": {"name":"daisy-776f7959cb-mysql","namespace":"kubeblocks-cloud-ns"}}