search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.01k stars 165 forks source link

[BUG] cluster role is none upgrade kb from 0.8.3 to 0.9 #7550

Closed JashBook closed 2 months ago

JashBook commented 2 months ago

Describe the bug A clear and concise description of what the bug is.

To Reproduce Steps to reproduce the behavior:

  1. install kb 0.8.3 
    kbcli kubeblocks install --create-namespace --version 0.8.3 --namespace kb-rgpamf
  2. create cluster
  3. upgrade kb to 0.9 
    kbcli kubeblocks upgrade --auto-approve  --set upgradeAddons=true --version 0.9.0-beta.32  --namespace kb-rgpamf
  4. See error 
    
kbcli cluster list-instances mysql-dngphn 
NAME                   NAMESPACE   CLUSTER        COMPONENT   STATUS    ROLE       ACCESSMODE   AZ              CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE    NODE                                                               CREATED-TIME                 
mysql-dngphn-mysql-0   default     mysql-dngphn   mysql       Running   follower   <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:1Gi   gke-cicd-gke-4mz6fyg-cicd-gke-4mz6fyg-3b1887dc-jsqv/10.128.0.118   Jun 16,2024 00:13 UTC+0800   
mysql-dngphn-mysql-1   default     mysql-dngphn   mysql       Running   <none>     <none>       us-central1-b   100m / 100m          512Mi / 512Mi           data:1Gi   gke-cicd-gke-4mz6fyg-cicd-gke-4mz6fyg-7d4255d8-d44n/10.128.0.61    Jun 18,2024 08:13 UTC+0800   
mysql-dngphn-mysql-2   default     mysql-dngphn   mysql       Running   leader     <none>       us-central1-a   100m / 100m          512Mi / 512Mi           data:1Gi   gke-cicd-gke-4mz6fyg-cicd-gke-4mz6fyg-9423bbe8-qbk1/10.128.0.46    Jun 14,2024 16:13 UTC+0800   
➜  ~

kbcli cluster list-instances postgres-jtgqtg NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME
postgres-jtgqtg-postgresql-0 default postgres-jtgqtg postgresql Running us-central1-b 100m / 100m 512Mi / 512Mi data:3Gi gke-cicd-gke-4mz6fyg-cicd-gke-4mz6fyg-7d4255d8-d44n/10.128.0.61 Jun 18,2024 08:13 UTC+0800
postgres-jtgqtg-postgresql-1 default postgres-jtgqtg postgresql Running primary us-central1-c 100m / 100m 512Mi / 512Mi data:3Gi gke-cicd-gke-4mz6fyg-cicd-gke-

➜ ~ kbcli cluster list-instances redis-oigzqs NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME
redis-oigzqs-redis-0 default redis-oigzqs redis Running us-central1-b 100m / 100m 512Mi / 512Mi data:1Gi gke-cicd-gke-4mz6fyg-cicd-gke-4mz6fyg-7d4255d8-d44n/10.128.0.61 Jun 18,2024 08:13 UTC+0800
redis-oigzqs-redis-1 default redis-oigzqs redis Running secondary us-central1-c 100m / 100m 512Mi / 512Mi data:1Gi gke-cicd-gke-4mz6fyg-cicd-gke-4mz6fyg-3b1887dc-jsqv/10.128.0.118 Jun 16,2024 00:13 UTC+0800
redis-oigzqs-redis-sentinel-0 default redis-oigzqs redis-sentinel CrashLoopBackOff us-central1-a 100m / 100m 512Mi / 512Mi data:1Gi gke-cicd-gke-4mz6fyg-cicd-gke-4mz6fyg-9423bbe8-t4xt/10.128.0.43 Jun 18,2024 11:08 UTC+0800
redis-oigzqs-redis-sentinel-1 default redis-oigzqs redis-sentinel CrashLoopBackOff us-central1-b 100m / 100m 512Mi / 512Mi data:1Gi gke-cicd-gke-4mz6fyg-cicd-gke-4mz6fyg-7d4255d8-cq0k/10.128.0.66 Jun 18,2024 11:05 UTC+0800
redis-oigzqs-redis-sentinel-2 default redis-oigzqs redis-sentinel CrashLoopBackOff us-central1-c 100m / 100m 512Mi / 512Mi data:1Gi gke-cicd-gke-4mz6fyg-cicd-gke-4mz6fyg-3b1887dc-jsqv/10.128.0.118 Jun 18,2024 11:08 UTC+0800
redis-oigzqs-redis-twemproxy-0 default redis-oigzqs redis-twemproxy Running us-central1-c 100m / 100m 512Mi / 512Mi data:1Gi gke-cicd-gke-4mz6fyg-cicd-gke-4mz6fyg-3b1887dc-n6zj/10.128.0.94 Jun 14,2024 21:00 UTC+0800
redis-oigzqs-redis-twemproxy-1 default redis-oigzqs redis-twemproxy Running us-central1-b 100m / 100m 512Mi / 512Mi data:1Gi gke-cicd-gke-4mz6fyg-cicd-gke-4mz6fyg-7d4255d8-d44n/10.128.0.61 Jun 18,2024 08:13 UTC+0800
redis-oigzqs-redis-twemproxy-2 default redis-oigzqs redis-twemproxy Running us-central1-a 100m / 100m 512Mi / 512Mi data:1Gi gke-cicd-gke-4mz6fyg-cicd-gke-4mz6fyg-9423bbe8-66sx/10.128.0.39 Jun 18,2024 09:53 UTC+0800
➜ ~ 

describe pod

kubectl describe pod mysql-dngphn-mysql-1 Name: mysql-dngphn-mysql-1 Namespace: default Priority: 0 Node: gke-cicd-gke-4mz6fyg-cicd-gke-4mz6fyg-7d4255d8-d44n/10.128.0.61 Start Time: Tue, 18 Jun 2024 08:13:22 +0800 Labels: app.kubernetes.io/component=mysql app.kubernetes.io/instance=mysql-dngphn app.kubernetes.io/managed-by=kubeblocks app.kubernetes.io/name=apecloud-mysql app.kubernetes.io/version= apps.kubeblocks.io/component-name=mysql controller-revision-hash=5b5bbb88f6 statefulset.kubernetes.io/pod-name=mysql-dngphn-mysql-1 workloads.kubeblocks.io/instance=mysql-dngphn-mysql workloads.kubeblocks.io/managed-by=InstanceSet Annotations: Status: Running IP: 10.40.54.20 IPs: IP: 10.40.54.20 Controlled By: StatefulSet/mysql-dngphn-mysql Containers: mysql: Container ID: containerd://6f7a5b0f9c664474ad61dd6253676ecc262b1449577ac8cca8adf1e0016b7f99 Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server:8.0.30-5.beta3.20240330.g94d1caf.15 Image ID: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-server@sha256:6e2c0daa38d2fc38d38c702d27eca1ec89d18eb3a15a349acadebed8661358eb Ports: 3306/TCP, 13306/TCP Host Ports: 0/TCP, 0/TCP Command: /scripts/setup.sh State: Running Started: Tue, 18 Jun 2024 08:15:48 +0800 Ready: True Restart Count: 0 Limits: cpu: 100m memory: 512Mi Requests: cpu: 100m memory: 512Mi Environment Variables from: mysql-dngphn-mysql-env ConfigMap Optional: false mysql-dngphn-mysql-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: mysql-dngphn-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).mysql-dngphn-mysql-headless.$(KB_NAMESPACE).svc SERVICE_PORT: 3306 MYSQL_ROOT_HOST: % MYSQL_ROOT_USER: <set to the key 'username' in secret 'mysql-dngphn-conn-credential'> Optional: false MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'mysql-dngphn-conn-credential'> Optional: false MYSQL_DATABASE: mydb MYSQL_USER: u1 MYSQL_PASSWORD: u1 CLUSTER_ID: 1 CLUSTER_START_INDEX: 1 REPLICATION_USER: replicator REPLICATION_PASSWORD:
MYSQL_TEMPLATE_CONFIG:
MYSQL_CUSTOM_CONFIG:
MYSQL_DYNAMIC_CONFIG:
KB_EMBEDDED_WESQL: 1 Mounts: /data/mysql from data (rw) /etc/annotations from annotations (rw) /opt/mysql from mysql-config (rw) /scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tnjl6 (ro) metrics: Container ID: containerd://94bdde4013c4b53e1425e017611c244a9da60da580b6ede76e35179f1c60ed16 Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto:0.1.2-beta.1 Image ID: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/agamotto@sha256:cbab349b90490807a8d5039bf01bc7e37334f20c98c7dd75bc7fc4cf9e5b10ee Port: 9104/TCP Host Port: 0/TCP Command: /scripts/agamotto.sh State: Running Started: Tue, 18 Jun 2024 08:15:51 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: mysql-dngphn-mysql-env ConfigMap Optional: false mysql-dngphn-mysql-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: mysql-dngphn-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).mysql-dngphn-mysql-headless.$(KB_NAMESPACE).svc DB_TYPE: MySQL ENDPOINT: localhost:3306 MYSQL_USER: <set to the key 'username' in secret 'mysql-dngphn-conn-credential'> Optional: false MYSQL_PASSWORD: <set to the key 'password' in secret 'mysql-dngphn-conn-credential'> Optional: false Mounts: /data/mysql from data (rw) /opt/agamotto from agamotto-configuration (rw) /scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tnjl6 (ro) vttablet: Container ID: containerd://f7795cdf1de28827889d966635d3566d62ff023190b3f591c25d868879a12846 Image: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale:0.2.6 Image ID: infracreate-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/apecloud-mysql-scale@sha256:131853e6f52acb0fd9d8715d022aaedc5373d87f71db5b5677fde42434807013 Ports: 15100/TCP, 16100/TCP Host Ports: 0/TCP, 0/TCP Command: /scripts/vttablet.sh State: Running Started: Tue, 18 Jun 2024 08:16:35 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: mysql-dngphn-mysql-env ConfigMap Optional: false mysql-dngphn-mysql-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: mysql-dngphn-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).mysql-dngphn-mysql-headless.$(KB_NAMESPACE).svc CELL: zone1 ETCD_SERVER: $(KB_CLUSTER_NAME)-vtcontroller-headless ETCD_PORT: 2379 TOPOLOGY_FLAGS: --topo_implementation etcd2 --topo_global_server_address $(ETCD_SERVER):$(ETCD_PORT) --topo_global_root /vitess/global VTTABLET_PORT: 15100 VTTABLET_GRPC_PORT: 16100 VTCTLD_HOST: $(KB_CLUSTER_NAME)-vtcontroller-headless VTCTLD_WEB_PORT: 15000 SERVICE_PORT: $(VTTABLET_PORT) MYSQL_ROOT_USER: <set to the key 'username' in secret 'mysql-dngphn-conn-credential'> Optional: false MYSQL_ROOT_PASSWORD: <set to the key 'password' in secret 'mysql-dngphn-conn-credential'> Optional: false Mounts: /conf from mysql-scale-config (rw) /scripts from scripts (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tnjl6 (ro) /vtdataroot from data (rw) lorry: Container ID: containerd://6f3dd37c1e8dd805f6cad1b60fa7b60d4fe30ce689800889c799d1be4ce665ad Image: docker.io/apecloud/kubeblocks-tools:0.8.3 Image ID: docker.io/apecloud/kubeblocks-tools@sha256:d0fe1c2841a637119ca7eb90e4a8de8ace06afd3b9dc08c5e657821005808916 Ports: 3501/TCP, 50001/TCP Host Ports: 0/TCP, 0/TCP Command: lorry --port 3501 --grpcport 50001 State: Running Started: Tue, 18 Jun 2024 08:16:35 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Readiness: http-get http://:3501/v1.0/checkrole delay=0s timeout=1s period=1s #success=1 #failure=3 Startup: tcp-socket :3501 delay=0s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: mysql-dngphn-mysql-env ConfigMap Optional: false mysql-dngphn-mysql-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: mysql-dngphn-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).mysql-dngphn-mysql-headless.$(KB_NAMESPACE).svc KB_BUILTIN_HANDLER: wesql KB_SERVICE_USER: <set to the key 'username' in secret 'mysql-dngphn-conn-credential'> Optional: false KB_SERVICE_PASSWORD: <set to the key 'password' in secret 'mysql-dngphn-conn-credential'> Optional: false KB_SERVICE_PORT: 3306 KB_DATA_PATH: /data/mysql KB_RSM_ACTION_SVC_LIST: null KB_RSM_ROLE_UPDATE_MECHANISM: DirectAPIServerEventUpdate KB_RSM_ROLE_PROBE_TIMEOUT: 1 KB_CLUSTER_NAME: (v1:metadata.labels['app.kubernetes.io/instance']) KB_COMP_NAME: (v1:metadata.labels['apps.kubeblocks.io/component-name']) KB_SERVICE_CHARACTER_TYPE: wesql Mounts: /data/mysql from data (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tnjl6 (ro) config-manager: Container ID: containerd://f4b77514e1db78d6b4329dfef066236b7389c8a5ae8c6625ec91e716e9451b02 Image: docker.io/apecloud/kubeblocks-tools:0.8.3 Image ID: docker.io/apecloud/kubeblocks-tools@sha256:d0fe1c2841a637119ca7eb90e4a8de8ace06afd3b9dc08c5e657821005808916 Port: Host Port: Command: env Args: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$(TOOLS_PATH) /bin/reloader --log-level info --operator-update-enable --tcp 9901 --config /opt/config-manager/config-manager.yaml State: Running Started: Tue, 18 Jun 2024 08:16:35 +0800 Ready: True Restart Count: 0 Limits: cpu: 0 memory: 0 Requests: cpu: 0 memory: 0 Environment Variables from: mysql-dngphn-mysql-env ConfigMap Optional: false mysql-dngphn-mysql-rsm-env ConfigMap Optional: false Environment: KB_POD_NAME: mysql-dngphn-mysql-1 (v1:metadata.name) KB_POD_UID: (v1:metadata.uid) KB_NAMESPACE: default (v1:metadata.namespace) KB_SA_NAME: (v1:spec.serviceAccountName) KB_NODENAME: (v1:spec.nodeName) KB_HOST_IP: (v1:status.hostIP) KB_POD_IP: (v1:status.podIP) KB_POD_IPS: (v1:status.podIPs) KB_HOSTIP: (v1:status.hostIP) KB_PODIP: (v1:status.podIP) KB_PODIPS: (v1:status.podIPs) KB_POD_FQDN: $(KB_POD_NAME).mysql-dngphn-mysql-headless.$(KB_NAMESPACE).svc CONFIG_MANAGER_POD_IP: (v1:status.podIP) DB_TYPE: mysql MYSQL_USER: <set to the key 'username' in secret 'mysql-dngphn-conn-credential'> Optional: false MYSQL_PASSWORD: <set to the key 'password' in secret 'mysql-dngphn-conn-credential'> Optional: false DATA_SOURCE_NAME: $(MYSQL_USER):$(MYSQL_PASSWORD)@(localhost:3306)/ TOOLS_PATH: /opt/kb-tools/reload/mysql-consensusset-config:/opt/config-manager Mounts: /conf from mysql-scale-config (rw) /opt/config-manager from config-manager-config (rw) /opt/kb-tools/reload/mysql-consensusset-config from cm-script-mysql-consensusset-config (rw) /opt/mysql from mysql-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-tnjl6 (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: data: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: data-mysql-dngphn-mysql-1 ReadOnly: false annotations: Type: DownwardAPI (a volume populated by information about the pod) Items: metadata.annotations['cs.apps.kubeblocks.io/leader'] -> leader metadata.annotations['apps.kubeblocks.io/component-replicas'] -> component-replicas agamotto-configuration: Type: ConfigMap (a volume populated by a ConfigMap) Name: mysql-dngphn-mysql-agamotto-configuration Optional: false scripts: Type: ConfigMap (a volume populated by a ConfigMap) Name: mysql-dngphn-mysql-apecloud-mysql-scripts Optional: false mysql-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: mysql-dngphn-mysql-mysql-consensusset-config Optional: false mysql-scale-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: mysql-dngphn-mysql-vttablet-config Optional: false cm-script-mysql-consensusset-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-mysql-reload-script-mysql-dngphn Optional: false config-manager-config: Type: ConfigMap (a volume populated by a ConfigMap) Name: sidecar-mysql-dngphn-mysql-config-manager-config Optional: false kube-api-access-tnjl6: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: Burstable Node-Selectors: Tolerations: kb-data=true:NoSchedule node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: 

logs apecloud mysql pod

kubectl logs mysql-dngphn-mysql-1 lorry 2024-06-18T00:16:35Z INFO Initialize DB manager 2024-06-18T00:16:35Z INFO KB_WORKLOAD_TYPE ENV not set 2024-06-18T00:16:35Z INFO HA HA starting 2024-06-18T00:16:35Z INFO Volume-Protection succeed to init volume protection {"pod": "mysql-dngphn-mysql-1", "spec": {"highWatermark":"0","volumes":[]}} 2024-06-18T00:16:35Z INFO HTTPServer Starting HTTP Server 2024-06-18T00:16:35Z INFO cronjobs env is not set {"env": "KB_CRON_JOBS"} 2024-06-18T00:16:35Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=mysql-dngphn,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql 2024-06-18T00:16:36Z INFO DCS-K8S podlist: 3 2024-06-18T00:16:36Z INFO pinger Waiting for dns resolution to be ready 2024-06-18T00:16:36Z INFO pinger dns check failed {"error": "new pinger failed: lookup mysql-dngphn-mysql-1.mysql-dngphn-mysql-headless.default.svc on 10.43.240.10:53: no such host"} 2024-06-18T00:16:38Z INFO WeSQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-18T00:16:38Z INFO WeSQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-18T00:16:39Z INFO pinger ping failed, socket: permission denied, but temporarily return true 2024-06-18T00:16:40Z INFO WeSQL DB startup ready 2024-06-18T00:16:40Z INFO event send event: map[event:Success operation:checkRole originalRole: role:Follower] 2024-06-18T00:16:40Z INFO event send event success {"message": "{\"event\":\"Success\",\"operation\":\"checkRole\",\"originalRole\":\"\",\"role\":\"Follower\"}"} 2024-06-18T00:16:49Z INFO pinger dns resolution is ready {"dns": "mysql-dngphn-mysql-1.mysql-dngphn-mysql-headless.default.svc"} 2024-06-18T00:16:49Z INFO HA cluster: &{mysql-dngphn-mysql default 3 0xc0006d9c00 0xc0006f0500 [{ mysql-dngphn-mysql-0 follower 10.40.34.15 3306 3501 3ea5caaf-4105-4cc5-81c2-9072c3ac2102 false 0xc000738480 mysql} { mysql-dngphn-mysql-1 3306 3501 ac1a35d0-bf3c-4bb2-b211-ba2f74891f17 false 0xc000739200 mysql} { mysql-dngphn-mysql-2 leader 10.40.8.10 3306 3501 77f0b21c-5680-4a09-9298-cd612e406966 false 0xc000739b00 mysql}] map[] 0xc00071a000} 2024-06-18T00:16:49Z INFO HA The database cluster is initialized. 2024-06-18T00:16:49Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:16:59Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:17:09Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:17:19Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:17:29Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:17:39Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:17:49Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:17:59Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:18:09Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:18:19Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:18:29Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:18:39Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:18:49Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:18:59Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:19:09Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:19:19Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:19:29Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:19:39Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"} 2024-06-18T00:19:49Z INFO WeSQL current member still follow the leader {"leader name": "mysql-dngphn-mysql-2"}


logs pg

k logs postgres-jtgqtg-postgresql-0 lorry 2024-06-18T00:16:20Z INFO Initialize DB manager 2024-06-18T00:16:20Z INFO KB_WORKLOAD_TYPE ENV not set 2024-06-18T00:16:20Z INFO Volume-Protection succeed to init volume protection {"pod": "postgres-jtgqtg-postgresql-0", "spec": {"highWatermark":"0","volumes":[]}} 2024-06-18T00:16:20Z INFO HTTPServer Starting HTTP Server 2024-06-18T00:16:20Z INFO cronjobs env is not set {"env": "KB_CRON_JOBS"} 2024-06-18T00:16:24Z ERROR PostgreSQL DB is not ready, ping failed {"error": "failed to connect to host=localhost user=postgres database=postgres: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"} github.com/apecloud/kubeblocks/pkg/lorry/engines/postgres.(Manager).IsPgReady /src/pkg/lorry/engines/postgres/manager.go:183 github.com/apecloud/kubeblocks/pkg/lorry/engines/postgres/officalpostgres.(Manager).IsDBStartupReady /src/pkg/lorry/engines/postgres/officalpostgres/manager.go:185 github.com/apecloud/kubeblocks/pkg/lorry/operations/replica.(CheckRole).Do /src/pkg/lorry/operations/replica/checkrole.go:129 github.com/apecloud/kubeblocks/pkg/lorry/httpserver.(api).RegisterOperations.OperationWrapper.func1 /src/pkg/lorry/httpserver/apis.go:119 github.com/fasthttp/router.(Router).Handler /go/pkg/mod/github.com/fasthttp/router@v1.4.20/router.go:420 github.com/apecloud/kubeblocks/pkg/lorry/httpserver.(server).StartNonBlocking.(server).apiLogger.func2 /src/pkg/lorry/httpserver/server.go:120 github.com/valyala/fasthttp.(Server).serveConn /go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/server.go:2359 github.com/valyala/fasthttp.(workerPool).workerFunc /go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/workerpool.go:224 github.com/valyala/fasthttp.(workerPool).getCh.func1 /go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/workerpool.go:196 2024-06-18T00:16:25Z ERROR PostgreSQL DB is not ready, ping failed {"error": "failed to connect to host=localhost user=postgres database=postgres: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"} github.com/apecloud/kubeblocks/pkg/lorry/engines/postgres.(Manager).IsPgReady /src/pkg/lorry/engines/postgres/manager.go:183 github.com/apecloud/kubeblocks/pkg/lorry/engines/postgres/officalpostgres.(Manager).IsDBStartupReady /src/pkg/lorry/engines/postgres/officalpostgres/manager.go:185 github.com/apecloud/kubeblocks/pkg/lorry/operations/replica.(CheckRole).Do /src/pkg/lorry/operations/replica/checkrole.go:129 github.com/apecloud/kubeblocks/pkg/lorry/httpserver.(api).RegisterOperations.OperationWrapper.func1 /src/pkg/lorry/httpserver/apis.go:119 github.com/fasthttp/router.(Router).Handler /go/pkg/mod/github.com/fasthttp/router@v1.4.20/router.go:420 github.com/apecloud/kubeblocks/pkg/lorry/httpserver.(server).StartNonBlocking.(server).apiLogger.func2 /src/pkg/lorry/httpserver/server.go:120 github.com/valyala/fasthttp.(Server).serveConn /go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/server.go:2359 github.com/valyala/fasthttp.(workerPool).workerFunc /go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/workerpool.go:224 github.com/valyala/fasthttp.(workerPool).getCh.func1 /go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/workerpool.go:196 ... 2024-06-18T00:16:46Z ERROR PostgreSQL DB is not ready, ping failed {"error": "failed to connect to host=localhost user=postgres database=postgres: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"} github.com/apecloud/kubeblocks/pkg/lorry/engines/postgres.(Manager).IsPgReady /src/pkg/lorry/engines/postgres/manager.go:183 github.com/apecloud/kubeblocks/pkg/lorry/engines/postgres/officalpostgres.(Manager).IsDBStartupReady /src/pkg/lorry/engines/postgres/officalpostgres/manager.go:185 github.com/apecloud/kubeblocks/pkg/lorry/operations/replica.(CheckRole).Do /src/pkg/lorry/operations/replica/checkrole.go:129 github.com/apecloud/kubeblocks/pkg/lorry/httpserver.(api).RegisterOperations.OperationWrapper.func1 /src/pkg/lorry/httpserver/apis.go:119 github.com/fasthttp/router.(Router).Handler /go/pkg/mod/github.com/fasthttp/router@v1.4.20/router.go:420 github.com/apecloud/kubeblocks/pkg/lorry/httpserver.(server).StartNonBlocking.(server).apiLogger.func2 /src/pkg/lorry/httpserver/server.go:120 github.com/valyala/fasthttp.(Server).serveConn /go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/server.go:2359 github.com/valyala/fasthttp.(workerPool).workerFunc /go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/workerpool.go:224 github.com/valyala/fasthttp.(workerPool).getCh.func1 /go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/workerpool.go:196 2024-06-18T00:16:47Z ERROR PostgreSQL DB is not ready, ping failed {"error": "failed to connect to host=localhost user=postgres database=postgres: dial error (dial tcp [::1]:5432: connect: cannot assign requested address)"} github.com/apecloud/kubeblocks/pkg/lorry/engines/postgres.(Manager).IsPgReady /src/pkg/lorry/engines/postgres/manager.go:183 github.com/apecloud/kubeblocks/pkg/lorry/engines/postgres/officalpostgres.(Manager).IsDBStartupReady /src/pkg/lorry/engines/postgres/officalpostgres/manager.go:185 github.com/apecloud/kubeblocks/pkg/lorry/operations/replica.(CheckRole).Do /src/pkg/lorry/operations/replica/checkrole.go:129 github.com/apecloud/kubeblocks/pkg/lorry/httpserver.(api).RegisterOperations.OperationWrapper.func1 /src/pkg/lorry/httpserver/apis.go:119 github.com/fasthttp/router.(Router).Handler /go/pkg/mod/github.com/fasthttp/router@v1.4.20/router.go:420 github.com/apecloud/kubeblocks/pkg/lorry/httpserver.(server).StartNonBlocking.(server).apiLogger.func2 /src/pkg/lorry/httpserver/server.go:120 github.com/valyala/fasthttp.(Server).serveConn /go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/server.go:2359 github.com/valyala/fasthttp.(workerPool).workerFunc /go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/workerpool.go:224 github.com/valyala/fasthttp.(workerPool).getCh.func1 /go/pkg/mod/github.com/valyala/fasthttp@v1.50.0/workerpool.go:196 2024-06-18T00:16:48Z INFO PostgreSQL DB startup ready 2024-06-18T00:16:48Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=postgres-jtgqtg,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=postgresql 2024-06-18T00:16:49Z INFO DCS-K8S podlist: 2 2024-06-18T00:16:49Z INFO DCS-K8S Leader configmap is not found {"configmap": "postgres-jtgqtg-postgresql-leader"} 2024-06-18T00:16:49Z INFO event send event: map[event:Success operation:checkRole originalRole: role:secondary] 2024-06-18T00:16:49Z INFO event send event success {"message": "{\"event\":\"Success\",\"operation\":\"checkRole\",\"originalRole\":\"\",\"role\":\"secondary\"}"} ➜ ~ 

logs redis

k logs redis-oigzqs-redis-0 lorry 2024-06-18T00:14:56Z INFO Initialize DB manager 2024-06-18T00:14:56Z INFO KB_WORKLOAD_TYPE ENV not set 2024-06-18T00:14:56Z INFO Volume-Protection succeed to init volume protection {"pod": "redis-oigzqs-redis-0", "spec": {"highWatermark":"0","volumes":[]}} 2024-06-18T00:14:56Z INFO HTTPServer Starting HTTP Server 2024-06-18T00:14:56Z INFO cronjobs env is not set {"env": "KB_CRON_JOBS"} 2024-06-18T00:15:02Z INFO Redis DB startup ready 2024-06-18T00:15:02Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=redis-oigzqs,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=redis 2024-06-18T00:15:03Z INFO DCS-K8S podlist: 2 2024-06-18T00:15:03Z INFO DCS-K8S Leader configmap is not found {"configmap": "redis-oigzqs-redis-leader"} 2024-06-18T00:15:03Z INFO event send event: map[event:Success operation:checkRole originalRole: role:primary] 2024-06-18T00:15:03Z INFO event send event success {"message": "{\"event\":\"Success\",\"operation\":\"checkRole\",\"originalRole\":\"\",\"role\":\"primary\"}"} 2024-06-18T00:15:08Z INFO event send event: map[event:Success operation:checkRole originalRole:primary role:secondary] 2024-06-18T00:15:08Z INFO event send event success {"message": "{\"event\":\"Success\",\"operation\":\"checkRole\",\"originalRole\":\"primary\",\"role\":\"secondary\"}"}



**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.
free6om commented 2 months ago

this issue is caused by a failure upgrade which been fixed by #7546