search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.14k stars 176 forks source link

[BUG] mysql cluster role is not correct upgrade from kb 0.8.3 to 0.9.0 #7518

Closed JashBook closed 4 months ago

JashBook commented 5 months ago

Describe the bug

kbcli version
Kubernetes: v1.29.4-gke.1043002
KubeBlocks: 0.9.0-beta.32
kbcli: 0.9.0-beta.24

To Reproduce Steps to reproduce the behavior:

  1. install kubeblocks 0.8.3
  2. create mysql cluster 
    apiVersion: apps.kubeblocks.io/v1alpha1
kind: Cluster
metadata:
labels:
clusterdefinition.kubeblocks.io/name: mysql
clusterversion.kubeblocks.io/name: mysql-5.7.44
name: asm-zquwfv
namespace: ns-zquwfv
spec:
affinity:
podAntiAffinity: Preferred
tenancy: SharedNode
clusterDefinitionRef: mysql
clusterVersionRef: mysql-5.7.44
componentSpecs:
- componentDefRef: mysql
enabledLogs:
- auditlog
- error
- slow
monitor: false
name: mysql
replicas: 2
resources:
  limits:
    cpu: 100m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 512Mi
serviceAccountName: kb-asm-zquwfv
switchPolicy:
  type: Noop
volumeClaimTemplates:
- name: data
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
terminationPolicy: Halt
  3. upgrade kubeblocks to 0.9.0
  4. See error 
    
kubectl get cluster -n ns-zquwfv  asm-zquwfv       
NAME         CLUSTER-DEFINITION   VERSION        TERMINATION-POLICY   STATUS     AGE
asm-zquwfv   mysql                mysql-5.7.44   Halt                 Updating   5h42m

kubectl get pod -n ns-zquwfv
NAME READY STATUS RESTARTS AGE asm-zquwfv-mysql-0 4/4 Running 2 (25m ago) 132m asm-zquwfv-mysql-1 4/4 Running 2 (25m ago) 5h31m

kbcli cluster list-instances -n ns-zquwfv asm-zquwfv NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME
asm-zquwfv-mysql-0 ns-zquwfv asm-zquwfv mysql Running secondary us-central1-b 100m / 100m 512Mi / 512Mi data:10Gi gke-cicd-gke-2vijq48-cicd-gke-2vijq48-71ad2b92-lqjg/10.128.0.15 Jun 12,2024 23:20 UTC+0800
asm-zquwfv-mysql-1 ns-zquwfv asm-zquwfv mysql Running secondary us-central1-c 100m / 100m 512Mi / 512Mi data:10Gi gke-cicd-gke-2vijq48-cicd-gke-2vijq48-f596aa9d-b85w/10.128.15.192 Jun 12,2024 20:02 UTC+0800 

logs pod

kubectl logs -n ns-zquwfv asm-zquwfv-mysql-0 lorry 2024-06-12T17:08:33Z INFO Initialize DB manager 2024-06-12T17:08:33Z INFO KB_WORKLOAD_TYPE ENV not set 2024-06-12T17:08:33Z INFO Volume-Protection succeed to init volume protection {"pod": "asm-zquwfv-mysql-0", "spec": {"highWatermark":"0","volumes":[]}} 2024-06-12T17:08:33Z INFO HTTPServer Starting HTTP Server 2024-06-12T17:08:33Z INFO HTTPServer API route path {"method": "POST", "path": ["/v1.0/deleteuser", "/v1.0/unlockinstance", "/v1.0/dataload", "/v1.0/getlag", "/v1.0/joinmember", "/v1.0/leavemember", "/v1.0/checkrunning", "/v1.0/createuser", "/v1.0/revokeuserrole", "/v1.0/lockinstance", "/v1.0/switchover", "/v1.0/rebuild", "/v1.0/exec", "/v1.0/grantuserrole", "/v1.0/volumeprotection", "/v1.0/postprovision", "/v1.0/preterminate", "/v1.0/datadump"]} 2024-06-12T17:08:33Z INFO HTTPServer API route path {"method": "GET", "path": ["/v1.0/query", "/v1.0/listsystemaccounts", "/v1.0/checkrole", "/v1.0/getrole", "/v1.0/describeuser", "/v1.0/listusers", "/v1.0/healthycheck"]} 2024-06-12T17:08:33Z INFO cronjobs env is not set {"env": "KB_CRON_JOBS"} 2024-06-12T17:08:33Z INFO MySQL DB startup ready 2024-06-12T17:08:34Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=asm-zquwfv,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql 2024-06-12T17:08:34Z INFO DCS-K8S podlist: 2 2024-06-12T17:08:34Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=asm-zquwfv,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql 2024-06-12T17:08:34Z INFO DCS-K8S podlist: 2 2024-06-12T17:08:34Z DEBUG checkrole check member {"member": "asm-zquwfv-mysql-0", "role": "primary"} 2024-06-12T17:08:34Z DEBUG checkrole check member {"member": "asm-zquwfv-mysql-1", "role": "secondary"} 2024-06-12T17:08:34Z INFO event send event: map[event:Success operation:checkRole originalRole:waitForStart role:{"term":"1718212114109210","PodRoleNamePairs":[{"podName":"asm-zquwfv-mysql-0","roleName":"primary","podUid":"436d148d-d6b0-4d36-83bb-6219908f989a"}]}] 2024-06-12T17:08:34Z INFO event send event success {"message": "{\"event\":\"Success\",\"operation\":\"checkRole\",\"originalRole\":\"waitForStart\",\"role\":\"{\\"term\\":\\"1718212114109210\\",\\"PodRoleNamePairs\\":[{\\"podName\\":\\"asm-zquwfv-mysql-0\\",\\"roleName\\":\\"primary\\",\\"podUid\\":\\"436d148d-d6b0-4d36-83bb-6219908f989a\\"}]}\"}"} 2024-06-12T17:08:40Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718212120 2024-06-12T17:08:40Z INFO MySQL cluster has no leader lease 2024-06-12T17:08:40Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=asm-zquwfv,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql 2024-06-12T17:08:40Z INFO DCS-K8S podlist: 2 2024-06-12T17:08:40Z DEBUG checkrole check member {"member": "asm-zquwfv-mysql-0", "role": "primary"} 2024-06-12T17:08:40Z DEBUG checkrole check member {"member": "asm-zquwfv-mysql-1", "role": "secondary"} 2024-06-12T17:08:40Z INFO checkrole there is a another leader {"member": "asm-zquwfv-mysql-1"} 2024-06-12T17:08:40Z INFO checkrole another leader's lorry is online, just ignore {"member": "asm-zquwfv-mysql-1"} 2024-06-12T17:08:40Z INFO event send event: map[event:Success operation:checkRole originalRole:primary role:{"term":"1718212120528802","PodRoleNamePairs":[{"podName":"asm-zquwfv-mysql-0","roleName":"secondary","podUid":"436d148d-d6b0-4d36-83bb-6219908f989a"}]}] 2024-06-12T17:08:40Z INFO event send event success {"message": "{\"event\":\"Success\",\"operation\":\"checkRole\",\"originalRole\":\"primary\",\"role\":\"{\\"term\\":\\"1718212120528802\\",\\"PodRoleNamePairs\\":[{\\"podName\\":\\"asm-zquwfv-mysql-0\\",\\"roleName\\":\\"secondary\\",\\"podUid\\":\\"436d148d-d6b0-4d36-83bb-6219908f989a\\"}]}\"}"} 2024-06-12T17:08:41Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718212121 2024-06-12T17:08:41Z INFO MySQL cluster has no leader lease 2024-06-12T17:08:42Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718212122 2024-06-12T17:08:42Z INFO MySQL cluster has no leader lease 2024-06-12T17:08:43Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718212123 2024-06-12T17:08:43Z INFO MySQL cluster has no leader lease 2024-06-12T17:08:44Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718212124 2024-06-12T17:08:44Z INFO MySQL cluster has no leader lease ... 2024-06-12T17:34:26Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718213666 2024-06-12T17:34:26Z INFO MySQL cluster has no leader lease 2024-06-12T17:34:27Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718213667 2024-06-12T17:34:27Z INFO MySQL cluster has no leader lease 2024-06-12T17:34:28Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718213668 2024-06-12T17:34:28Z INFO MySQL cluster has no leader lease

logs pod2

kubectl logs -n ns-zquwfv asm-zquwfv-mysql-1 lorry 2024-06-12T17:08:26Z INFO Initialize DB manager 2024-06-12T17:08:26Z INFO KB_WORKLOAD_TYPE ENV not set 2024-06-12T17:08:26Z INFO Volume-Protection succeed to init volume protection {"pod": "asm-zquwfv-mysql-1", "spec": {"highWatermark":"0","volumes":[]}} 2024-06-12T17:08:26Z INFO HTTPServer Starting HTTP Server 2024-06-12T17:08:26Z INFO HTTPServer API route path {"method": "GET", "path": ["/v1.0/getrole", "/v1.0/listusers", "/v1.0/healthycheck", "/v1.0/listsystemaccounts", "/v1.0/query", "/v1.0/describeuser", "/v1.0/checkrole"]} 2024-06-12T17:08:26Z INFO HTTPServer API route path {"method": "POST", "path": ["/v1.0/switchover", "/v1.0/grantuserrole", "/v1.0/preterminate", "/v1.0/rebuild", "/v1.0/unlockinstance", "/v1.0/joinmember", "/v1.0/getlag", "/v1.0/leavemember", "/v1.0/exec", "/v1.0/checkrunning", "/v1.0/dataload", "/v1.0/deleteuser", "/v1.0/lockinstance", "/v1.0/revokeuserrole", "/v1.0/volumeprotection", "/v1.0/postprovision", "/v1.0/datadump", "/v1.0/createuser"]} 2024-06-12T17:08:26Z INFO cronjobs env is not set {"env": "KB_CRON_JOBS"} 2024-06-12T17:08:31Z INFO MySQL DB startup ready 2024-06-12T17:08:31Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=asm-zquwfv,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql 2024-06-12T17:08:31Z INFO DCS-K8S podlist: 2 2024-06-12T17:08:32Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=asm-zquwfv,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql 2024-06-12T17:08:32Z INFO DCS-K8S podlist: 2 2024-06-12T17:08:32Z DEBUG checkrole check member {"member": "asm-zquwfv-mysql-0", "role": "primary"} 2024-06-12T17:08:32Z DEBUG checkrole check member {"member": "asm-zquwfv-mysql-1", "role": "primary"} 2024-06-12T17:08:32Z INFO event send event: map[event:Success operation:checkRole originalRole:waitForStart role:{"term":"1718212112083805","PodRoleNamePairs":[{"podName":"asm-zquwfv-mysql-1","roleName":"secondary","podUid":"0fff5f3e-f89f-42cd-b0fe-a1fa325f0dca"}]}] 2024-06-12T17:08:32Z INFO event send event success {"message": "{\"event\":\"Success\",\"operation\":\"checkRole\",\"originalRole\":\"waitForStart\",\"role\":\"{\\"term\\":\\"1718212112083805\\",\\"PodRoleNamePairs\\":[{\\"podName\\":\\"asm-zquwfv-mysql-1\\",\\"roleName\\":\\"secondary\\",\\"podUid\\":\\"0fff5f3e-f89f-42cd-b0fe-a1fa325f0dca\\"}]}\"}"} 2024-06-12T17:08:35Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718212115 2024-06-12T17:08:35Z INFO MySQL cluster has no leader lease 2024-06-12T17:08:36Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718212116 2024-06-12T17:08:36Z INFO MySQL cluster has no leader lease 2024-06-12T17:08:37Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718212117 2024-06-12T17:08:37Z INFO MySQL cluster has no leader lease 2024-06-12T17:08:38Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718212118 2024-06-12T17:08:38Z INFO MySQL cluster has no leader lease 2024-06-12T17:08:39Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718212119 2024-06-12T17:08:39Z INFO MySQL cluster has no leader lease ... 2024-06-12T17:35:36Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718213736 2024-06-12T17:35:36Z INFO MySQL cluster has no leader lease 2024-06-12T17:35:37Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718213737 2024-06-12T17:35:37Z INFO MySQL cluster has no leader lease 2024-06-12T17:35:38Z INFO DCS-K8S lock expired: map[acquire-time:1718205898 dbstate:{"OpTimestamp":1718212089,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4773,\nb0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778","gtid_purged":"","hostname":"asm-zquwfv-mysql-0","read_only":"0","server_uuid":"17cbe07b-28b2-11ef-88da-fe36919eb99b","super_read_only":"0"}} leader:asm-zquwfv-mysql-0 renew-time:1718212099 ttl:15], now: 1718213738 2024-06-12T17:35:38Z INFO MySQL cluster has no leader lease


connect pod 1

kbcli cluster connect -n ns-zquwfv -i asm-zquwfv-mysql-0 Defaulted container "mysql" out of: mysql, metrics, lorry, config-manager, init-data (init) mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 541 Server version: 5.7.44-log MySQL Community Server (GPL)

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show slave status\G; 1. row Slave_IO_State: Master_Host: asm-zquwfv-mysql-1.asm-zquwfv-mysql-headless Master_User: root Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000004 Read_Master_Log_Pos: 958726 Relay_Log_File: relay-bin.000006 Relay_Log_Pos: 414 Relay_Master_Log_File: mysql-bin.000004 Slave_IO_Running: No Slave_SQL_Running: No Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 958726 Relay_Log_Space: 615 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: NULL Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 2 Master_UUID: b0cd31bf-28b3-11ef-b31b-ee0c9d44b277 Master_Info_File: /var/lib/mysql/data/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: 17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4776, b0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: 1 row in set (0.01 sec)

ERROR: No query specified

mysql> mysql> show master status\G; 1. row File: mysql-bin.000006 Position: 506774 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 17cbe07b-28b2-11ef-88da-fe36919eb99b:1-4776, b0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-778 1 row in set (0.00 sec)

ERROR: No query specified

mysql> 


connect pod 2

kbcli cluster connect -n ns-zquwfv -i asm-zquwfv-mysql-1 Defaulted container "mysql" out of: mysql, metrics, lorry, config-manager, init-data (init) mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1359 Server version: 5.7.44-log MySQL Community Server (GPL)

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show slave status\G; 1. row Slave_IO_State: Master_Host: asm-zquwfv-mysql-0.asm-zquwfv-mysql-headless Master_User: root Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000003 Read_Master_Log_Pos: 794278 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 747152 Relay_Master_Log_File: mysql-bin.000003 Slave_IO_Running: No Slave_SQL_Running: No Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 794278 Relay_Log_Space: 747353 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: NULL Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 2003 Last_IO_Error: error reconnecting to master 'root@asm-zquwfv-mysql-0.asm-zquwfv-mysql-headless:3306' - retry-time: 60 retries: 1 Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: 17cbe07b-28b2-11ef-88da-fe36919eb99b Master_Info_File: /var/lib/mysql/data/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: 240612 14:33:50 Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: 17cbe07b-28b2-11ef-88da-fe36919eb99b:175-2916 Executed_Gtid_Set: 17cbe07b-28b2-11ef-88da-fe36919eb99b:1-2916, b0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-777 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: 1 row in set (0.00 sec)

ERROR: No query specified

mysql> show master status\G; 1. row File: mysql-bin.000004 Position: 958726 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 17cbe07b-28b2-11ef-88da-fe36919eb99b:1-2916, b0cd31bf-28b3-11ef-b31b-ee0c9d44b277:1-777 1 row in set (0.00 sec)

ERROR: No query specified



**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.
JashBook commented 5 months ago 
kbcli version
Kubernetes: v1.27.13-gke.1070000
KubeBlocks: 0.9.0-beta.33
kbcli: 0.9.0-beta.27

The problem can still be reproduced mysql and apecloud-mysql

kbcli cluster list-instances -n ns-cluster                   
NAME                                NAMESPACE    CLUSTER            COMPONENT          STATUS    ROLE        ACCESSMODE   AZ              CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)             STORAGE            NODE                                                               CREATED-TIME                 
asm-cluster-mysql-0                 ns-cluster   asm-cluster        mysql              Running   secondary   <none>       us-central1-b   200m / 200m          644245094400m / 644245094400m     data:10Gi          gke-cicd-gke-fdouqke-cicd-gke-fdouqke-bfc4490e-bhvn/10.128.0.95    Jun 19,2024 14:53 UTC+0800   
asm-cluster-mysql-1                 ns-cluster   asm-cluster        mysql              Running   secondary   <none>       us-central1-b   200m / 200m          644245094400m / 644245094400m     data:10Gi          gke-cicd-gke-fdouqke-cicd-gke-fdouqke-bfc4490e-4nkb/10.128.0.62    Jun 19,2024 14:52 UTC+0800   
asm-cluster-mysql-2                 ns-cluster   asm-cluster        mysql              Running   secondary   <none>       us-central1-b   200m / 200m          644245094400m / 644245094400m     data:10Gi          

mysql-cluster-mysql-1               ns-cluster   mysql-cluster      mysql              Running   follower    <none>       us-central1-b   200m / 200m          644245094400m / 644245094400m     data:1Gi           gke-cicd-gke-fdouqke-cicd-gke-fdouqke-bfc4490e-b19h/10.128.0.44    Jun 19,2024 17:14 UTC+0800   
mysql-cluster-mysql-2               ns-cluster   mysql-cluster      mysql              Running   follower    <none>       us-central1-b   200m / 200m          644245094400m / 644245094400m     data:1Gi           gke-cicd-gke-fdouqke-cicd-gke-fdouqke-bfc4490e-bhvn/10.128.0.95    Jun 19,2024 15:01 UTC+0800
xuriwuyun commented 4 months ago

The change of all replicas' roles to secondary is caused by a KB upgrade, but the MySQL addon was not upgraded. In release 0.9, the MySQL HA service was moved from Lorry to Syncer. When upgrading KB, it is necessary to update the addon and restart the cluster.

JashBook commented 4 months ago

The change of all replicas' roles to secondary is caused by a KB upgrade, but the MySQL addon was not upgraded. In release 0.9, the MySQL HA service was moved from Lorry to Syncer. When upgrading KB, it is necessary to update the addon and restart the cluster.

  1. upgrade addon 
    
kbcli addon upgrade  mysql  --force

kbcli addon list mysql NAME VERSION PROVIDER STATUS AUTO-INSTALL
mysql 0.9.1 community Enabled true 

2. force restart cluster

kbcli cluster restart -n ns-cluster asm-cluster --auto-approve --force

3. see error

kubectl get pod -n ns-cluster NAME READY STATUS RESTARTS AGE asm-cluster-mysql-0 4/4 Running 2 (36m ago) 158m asm-cluster-mysql-1 4/4 Running 0 10m

kbcli cluster list-instances asm-cluster -n ns-cluster NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME
asm-cluster-mysql-0 ns-cluster asm-cluster mysql Running secondary us-central1-a 200m / 200m 644245094400m / 644245094400m data:10Gi gke-cicd-gke-q3zypql-cicd-gke-q3zypql-771d13bc-9w0d/10.128.0.60 Jun 20,2024 12:42 UTC+0800
asm-cluster-mysql-1 ns-cluster asm-cluster mysql Running us-central1-a 200m / 200m 644245094400m / 644245094400m data:10Gi gke-cicd-gke-q3zypql-cicd-gke-q3zypql-771d13bc-7sd8/10.128.15.207 Jun 20,2024 15:10 UTC+0800 

logs pod lorry

kubectl logs -n ns-cluster asm-cluster-mysql-1 lorry 2024-06-20T07:12:12Z INFO Initialize DB manager 2024-06-20T07:12:12Z INFO KB_WORKLOAD_TYPE ENV not set 2024-06-20T07:12:12Z INFO Volume-Protection succeed to init volume protection {"pod": "asm-cluster-mysql-1", "spec": {"highWatermark":"0","volumes":[]}} 2024-06-20T07:12:12Z INFO HTTPServer Starting HTTP Server 2024-06-20T07:12:12Z INFO HTTPServer API route path {"method": "POST", "path": ["/v1.0/leavemember", "/v1.0/postprovision", "/v1.0/dataload", "/v1.0/joinmember", "/v1.0/rebuild", "/v1.0/createuser", "/v1.0/datadump", "/v1.0/deleteuser", "/v1.0/preterminate", "/v1.0/checkrunning", "/v1.0/getlag", "/v1.0/exec", "/v1.0/volumeprotection", "/v1.0/switchover", "/v1.0/lockinstance", "/v1.0/revokeuserrole", "/v1.0/grantuserrole", "/v1.0/unlockinstance"]} 2024-06-20T07:12:12Z INFO HTTPServer API route path {"method": "GET", "path": ["/v1.0/getrole", "/v1.0/describeuser", "/v1.0/listusers", "/v1.0/healthycheck", "/v1.0/listsystemaccounts", "/v1.0/checkrole", "/v1.0/query"]} 2024-06-20T07:12:12Z INFO cronjobs env is not set {"env": "KB_CRON_JOBS"} 2024-06-20T07:12:15Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:16Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:17Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:18Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:19Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:20Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:21Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:22Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:23Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:24Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:25Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:26Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:27Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:28Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:29Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"} 2024-06-20T07:12:30Z INFO MySQL DB is not ready {"error": "dial tcp 127.0.0.1:3306: connect: connection refused"}

logs pod-0 lorry

kubectl logs -n ns-cluster asm-cluster-mysql-0 lorry 2024-06-20T06:44:32Z INFO Initialize DB manager 2024-06-20T06:44:32Z INFO KB_WORKLOAD_TYPE ENV not set 2024-06-20T06:44:32Z INFO Volume-Protection succeed to init volume protection {"pod": "asm-cluster-mysql-0", "spec": {"highWatermark":"0","volumes":[]}} 2024-06-20T06:44:32Z INFO HTTPServer Starting HTTP Server 2024-06-20T06:44:32Z INFO HTTPServer API route path {"method": "POST", "path": ["/v1.0/rebuild", "/v1.0/datadump", "/v1.0/dataload", "/v1.0/createuser", "/v1.0/lockinstance", "/v1.0/unlockinstance", "/v1.0/checkrunning", "/v1.0/leavemember", "/v1.0/switchover", "/v1.0/preterminate", "/v1.0/getlag", "/v1.0/exec", "/v1.0/volumeprotection", "/v1.0/postprovision", "/v1.0/deleteuser", "/v1.0/revokeuserrole", "/v1.0/grantuserrole", "/v1.0/joinmember"]} 2024-06-20T06:44:32Z INFO HTTPServer API route path {"method": "GET", "path": ["/v1.0/describeuser", "/v1.0/healthycheck", "/v1.0/checkrole", "/v1.0/getrole", "/v1.0/listsystemaccounts", "/v1.0/query", "/v1.0/listusers"]} 2024-06-20T06:44:32Z INFO cronjobs env is not set {"env": "KB_CRON_JOBS"} 2024-06-20T06:44:36Z INFO MySQL DB startup ready 2024-06-20T06:44:36Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=asm-cluster,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql 2024-06-20T06:44:36Z INFO DCS-K8S podlist: 2 2024-06-20T06:44:36Z INFO DCS-K8S lock expired: map[acquire-time:1718858587 dbstate:{"OpTimestamp":1718865847,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"3ee71ae1-2eb9-11ef-b18e-e613871ad446:1-2929","gtid_purged":"","hostname":"asm-cluster-mysql-0","read_only":"0","server_uuid":"3ee71ae1-2eb9-11ef-b18e-e613871ad446","super_read_only":"0"}} leader:asm-cluster-mysql-0 renew-time:1718865857 ttl:15], now: 1718865876 2024-06-20T06:44:36Z INFO DCS-K8S lock expired: map[acquire-time:1718858587 dbstate:{"OpTimestamp":1718865847,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"3ee71ae1-2eb9-11ef-b18e-e613871ad446:1-2929","gtid_purged":"","hostname":"asm-cluster-mysql-0","read_only":"0","server_uuid":"3ee71ae1-2eb9-11ef-b18e-e613871ad446","super_read_only":"0"}} leader:asm-cluster-mysql-0 renew-time:1718865857 ttl:15], now: 1718865876 2024-06-20T06:44:36Z INFO MySQL cluster has no leader lease 2024-06-20T06:44:36Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=asm-cluster,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql 2024-06-20T06:44:36Z INFO DCS-K8S podlist: 2 2024-06-20T06:44:36Z DEBUG checkrole check member {"member": "asm-cluster-mysql-0", "role": "primary"} 2024-06-20T06:44:36Z DEBUG checkrole check member {"member": "asm-cluster-mysql-1", "role": "secondary"} 2024-06-20T06:44:36Z INFO checkrole there is a another leader {"member": "asm-cluster-mysql-1"} 2024-06-20T06:44:36Z INFO checkrole another leader's lorry is online, just ignore {"member": "asm-cluster-mysql-1"} 2024-06-20T06:44:36Z INFO event send event: map[event:Success operation:checkRole originalRole:waitForStart role:{"term":"1718865876914012","PodRoleNamePairs":[{"podName":"asm-cluster-mysql-0","roleName":"secondary","podUid":"6142379c-975a-4ef1-9867-f66ca657e67b"}]}] 2024-06-20T06:44:36Z INFO DCS-K8S lock expired: map[acquire-time:1718858587 dbstate:{"OpTimestamp":1718865847,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"3ee71ae1-2eb9-11ef-b18e-e613871ad446:1-2929","gtid_purged":"","hostname":"asm-cluster-mysql-0","read_only":"0","server_uuid":"3ee71ae1-2eb9-11ef-b18e-e613871ad446","super_read_only":"0"}} leader:asm-cluster-mysql-0 renew-time:1718865857 ttl:15], now: 1718865876 2024-06-20T06:44:36Z INFO MySQL cluster has no leader lease 2024-06-20T06:44:36Z INFO event send event success {"message": "{\"event\":\"Success\",\"operation\":\"checkRole\",\"originalRole\":\"waitForStart\",\"role\":\"{\\"term\\":\\"1718865876914012\\",\\"PodRoleNamePairs\\":[{\\"podName\\":\\"asm-cluster-mysql-0\\",\\"roleName\\":\\"secondary\\",\\"podUid\\":\\"6142379c-975a-4ef1-9867-f66ca657e67b\\"}]}\"}"} 2024-06-20T06:44:37Z INFO DCS-K8S lock expired: map[acquire-time:1718858587 dbstate:{"OpTimestamp":1718865847,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"3ee71ae1-2eb9-11ef-b18e-e613871ad446:1-2929","gtid_purged":"","hostname":"asm-cluster-mysql-0","read_only":"0","server_uuid":"3ee71ae1-2eb9-11ef-b18e-e613871ad446","super_read_only":"0"}} leader:asm-cluster-mysql-0 renew-time:1718865857 ttl:15], now: 1718865877 2024-06-20T06:44:37Z INFO MySQL cluster has no leader lease 2024-06-20T06:44:39Z INFO DCS-K8S lock expired: map[acquire-time:1718858587 dbstate:{"OpTimestamp":1718865847,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"3ee71ae1-2eb9-11ef-b18e-e613871ad446:1-2929","gtid_purged":"","hostname":"asm-cluster-mysql-0","read_only":"0","server_uuid":"3ee71ae1-2eb9-11ef-b18e-e613871ad446","super_read_only":"0"}} leader:asm-cluster-mysql-0 renew-time:1718865857 ttl:15], now: 1718865879 2024-06-20T06:44:39Z INFO MySQL cluster has no leader lease 2024-06-20T06:44:39Z INFO DCS-K8S lock expired: map[acquire-time:1718858587 dbstate:{"OpTimestamp":1718865847,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"3ee71ae1-2eb9-11ef-b18e-e613871ad446:1-2929","gtid_purged":"","hostname":"asm-cluster-mysql-0","read_only":"0","server_uuid":"3ee71ae1-2eb9-11ef-b18e-e613871ad446","super_read_only":"0"}} leader:asm-cluster-mysql-0 renew-time:1718865857 ttl:15], now: 1718865879 2024-06-20T06:44:39Z INFO MySQL cluster has no leader lease 2024-06-20T06:44:40Z INFO DCS-K8S lock expired: map[acquire-time:1718858587 dbstate:{"OpTimestamp":1718865847,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"3ee71ae1-2eb9-11ef-b18e-e613871ad446:1-2929","gtid_purged":"","hostname":"asm-cluster-mysql-0","read_only":"0","server_uuid":"3ee71ae1-2eb9-11ef-b18e-e613871ad446","super_read_only":"0"}} leader:asm-cluster-mysql-0 renew-time:1718865857 ttl:15], now: 1718865880 2024-06-20T06:44:40Z INFO MySQL cluster has no leader lease 2024-06-20T06:44:41Z INFO DCS-K8S lock expired: map[acquire-time:1718858587 dbstate:{"OpTimestamp":1718865847,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"3ee71ae1-2eb9-11ef-b18e-e613871ad446:1-2929","gtid_purged":"","hostname":"asm-cluster-mysql-0","read_only":"0","server_uuid":"3ee71ae1-2eb9-11ef-b18e-e613871ad446","super_read_only":"0"}} leader:asm-cluster-mysql-0 renew-time:1718865857 ttl:15], now: 1718865881 2024-06-20T06:44:41Z INFO MySQL cluster has no leader lease 2024-06-20T06:44:43Z INFO DCS-K8S lock expired: map[acquire-time:1718858587 dbstate:{"OpTimestamp":1718865847,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"3ee71ae1-2eb9-11ef-b18e-e613871ad446:1-2929","gtid_purged":"","hostname":"asm-cluster-mysql-0","read_only":"0","server_uuid":"3ee71ae1-2eb9-11ef-b18e-e613871ad446","super_read_only":"0"}} leader:asm-cluster-mysql-0 renew-time:1718865857 ttl:15], now: 1718865883 2024-06-20T06:44:43Z INFO MySQL cluster has no leader lease 2024-06-20T06:44:43Z INFO DCS-K8S lock expired: map[acquire-time:1718858587 dbstate:{"OpTimestamp":1718865847,"Extra":{"Binlog_File":"mysql-bin.000006","Binlog_Pos":"","gtid_executed":"3ee71ae1-2eb9-11ef-b18e-e613871ad446:1-2929","gtid_purged":"","hostname":"asm-cluster-mysql-0","read_only":"0","server_uuid":"3ee71ae1-2eb9-11ef-b18e-e613871ad446","super_read_only":"0"}} leader:asm-cluster-mysql-0 renew-time:1718865857 ttl:15], now: 1718865883 2024-06-20T06:44:43Z INFO MySQL cluster has no leader lease

ahjing99 commented 4 months ago

still failed when upgrade kb from 0.8.3 to 0.9

  1. 0n 0.8.3 
    k get cluster
NAME              CLUSTER-DEFINITION    VERSION                             TERMINATION-POLICY   STATUS    AGE
asmysql-zileht    mysql                 mysql-5.7.44                        Delete               Running   45m
  2. Upgrade kb 
    
➜  ~ kbcli kubeblocks upgrade --version 0.9.0-beta.41
Current KubeBlocks version 0.8.3.
Kubernetes version 1.29.4
Kubernetes provider GKE
kbcli version 0.9.0-beta.1
Upgrade KubeBlocks from 0.8.3 to 0.9.0-beta.41
Please type 'Yes/yes' to confirm your operation: yes
Add and update repo kubeblocks                     OK
Keep addons                                        OK
Stop KubeBlocks 0.8.3                              OK
Stop DataProtection                                OK
Conversion old version[0.8.3] CRs to new version[0.9.0-beta.41] OK
Upgrade CRDs                                       OK
update new version CRs                             OK
⣾ Upgrading KubeBlocks to 0.9.0-beta.41             E0628 15:28:06.963868   91761 request.go:1116] Unexpected error when reading response body: net/http: request canceled (Client.Timeout or context cancellation while reading body)
⣾ Upgrading KubeBlocks to 0.9.0-beta.41             E0628 15:28:56.378184   91761 request.go:1116] Unexpected error when reading response body: net/http: request canceled (Client.Timeout or context cancellation while reading body)
Upgrading KubeBlocks to 0.9.0-beta.41              OK

KubeBlocks has been upgraded to 0.9.0-beta.41 SUCCESSFULLY!

-> Basic commands for cluster: kbcli cluster create -h # help information about creating a database cluster kbcli cluster list # list all database clusters kbcli cluster describe # get cluster information

-> Uninstall KubeBlocks: kbcli kubeblocks uninstall

3. mysql is always updating for role is not correct

➜ ~ kbcli cluster describe asmysql-zileht Name: asmysql-zileht Created Time: Jun 28,2024 14:45 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY default mysql mysql-5.7.44 Updating Delete

Endpoints: COMPONENT MODE INTERNAL EXTERNAL mysql ReadWrite asmysql-zileht-mysql.default.svc.cluster.local:3306

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql asmysql-zileht-mysql-0 secondary Running us-central1-c gke-yijing-default-pool-b1c0b252-jcbk/10.128.15.249 Jun 28,2024 14:46 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql false 100m / 100m 512Mi / 512Mi data:10Gi kb-default-sc

Images: COMPONENT TYPE IMAGE mysql mysql docker.io/mysql:5.7.44

Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME

Show cluster events: kbcli cluster list-events -n default asmysql-zileht

4. upgrade addon and restart cluster still not work

➜ ~ kbcli addon upgrade mysql --force Warning: --force flag will skip version checks, which may result in the cluster not running correctly. Addon mysql-0.9.1 upgrade successed. ➜ ~ kbcli addon list mysql NAME VERSION PROVIDER STATUS AUTO-INSTALL mysql 0.9.1 community Enabled true ➜ ~ kbcli cluster restart asmysql-zileht --auto-approve --force OpsRequest asmysql-zileht-restart-pv5tc created successfully, you can view the progress: kbcli cluster describe-ops asmysql-zileht-restart-pv5tc -n default ➜ ~ ➜ ~ kbcli cluster describe asmysql-zileht Name: asmysql-zileht Created Time: Jun 28,2024 14:45 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY default mysql mysql-5.7.44 Updating Delete

Endpoints: COMPONENT MODE INTERNAL EXTERNAL mysql ReadWrite asmysql-zileht-mysql.default.svc.cluster.local:3306

Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql asmysql-zileht-mysql-0 Running us-central1-c gke-yijing-default-pool-b1c0b252-jcbk/10.128.15.249 Jun 28,2024 15:39 UTC+0800

Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql false 100m / 100m 512Mi / 512Mi data:10Gi kb-default-sc

Images: COMPONENT TYPE IMAGE mysql mysql docker.io/apecloud/mysql:5.7.44

Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME

Show cluster events: kbcli cluster list-events -n default asmysql-zileht

5. logs

➜ ~ k logs asmysql-zileht-mysql-0 Defaulted container "mysql" out of: mysql, metrics, lorry, config-manager, init-data (init), init-syncer (init), init-xtrabackup (init) 2024-06-28T07:40:30Z INFO Initialize DB manager 2024-06-28T07:40:30Z INFO KB_WORKLOAD_TYPE ENV not set 2024-06-28T07:40:30Z INFO HTTPServer Starting HTTP Server 2024-06-28T07:40:30Z INFO HTTPServer API route path {"method": "POST", "path": ["/v1.0/switchover", "/v1.0/rebuild", "/v1.0/start", "/v1.0/stop"]} 2024-06-28T07:40:30Z INFO HTTPServer API route path {"method": "GET", "path": ["/v1.0/datasync"]} 2024-06-28T07:40:30Z INFO HA HA starting 2024-06-28T07:40:30Z INFO pinger Waiting for dns resolution to be ready 2024-06-28T07:40:30Z INFO HTTPServer http server {"listen address": "0.0.0.0", "port": 3601} 2024-06-28T07:40:30Z INFO pinger dns resolution is ready {"dns": "asmysql-zileht-mysql-0.asmysql-zileht-mysql-headless.default.svc"} 2024-06-28T07:40:30Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=asmysql-zileht,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql 2024-06-28T07:40:30Z INFO DCS-K8S podlist: 1 2024-06-28T07:40:30Z INFO DCS-K8S lock expired: map[acquire-time:1719557264 dbstate:{"OpTimestamp":1719559925,"Extra":{"Binlog_File":"mysql-bin.000003","Binlog_Pos":"","gtid_executed":"466c1cc2-351a-11ef-a488-ba2e35ffc789:1-801","gtid_purged":"","hostname":"asmysql-zileht-mysql-0","read_only":"0","server_uuid":"466c1cc2-351a-11ef-a488-ba2e35ffc789","super_read_only":"0"}} extra: leader:asmysql-zileht-mysql-0 renew-time:1719559935 ttl:15], now: 1719560430 2024-06-28T07:40:31Z DEBUG HA cluster info {"cluster": {"ClusterCompName":"asmysql-zileht-mysql","Namespace":"default","Replicas":1,"HaConfig":{"ClusterInitializeOwner":"","DeleteMembers":{}},"Leader":{"DBState":{"OpTimestamp":1719559925,"Extra":{"Binlog_File":"mysql-bin.000003","Binlog_Pos":"","gtid_executed":"466c1cc2-351a-11ef-a488-ba2e35ffc789:1-801","gtid_purged":"","hostname":"asmysql-zileht-mysql-0","read_only":"0","server_uuid":"466c1cc2-351a-11ef-a488-ba2e35ffc789","super_read_only":"0"}},"Index":"47236","Name":"","AcquireTime":1719557264,"RenewTime":1719559935,"TTL":15,"Resource":{"metadata":{"name":"asmysql-zileht-mysql-leader","namespace":"default","uid":"84fb2ad3-65d5-45ed-a3b2-6b497c2163cd","resourceVersion":"47236","creationTimestamp":"2024-06-28T06:47:44Z","labels":{"app.kubernetes.io/instance":"asmysql-zileht","app.kubernetes.io/managed-by":"kubeblocks","apps.kubeblocks.io/component-name":"mysql"},"annotations":{"acquire-time":"1719557264","dbstate":"{\"OpTimestamp\":1719559925,\"Extra\":{\"Binlog_File\":\"mysql-bin.000003\",\"Binlog_Pos\":\"\",\"gtid_executed\":\"466c1cc2-351a-11ef-a488-ba2e35ffc789:1-801\",\"gtid_purged\":\"\",\"hostname\":\"asmysql-zileht-mysql-0\",\"read_only\":\"0\",\"server_uuid\":\"466c1cc2-351a-11ef-a488-ba2e35ffc789\",\"super_read_only\":\"0\"}}","extra":"","leader":"asmysql-zileht-mysql-0","renew-time":"1719559935","ttl":"15"},"ownerReferences":[{"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","name":"asmysql-zileht","uid":"6e19661b-3bd0-4cf2-a1b5-6c6dd211fa60"}],"managedFields":[{"manager":"lorry","operation":"Update","apiVersion":"v1","time":"2024-06-28T07:32:15Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:acquire-time":{},"f:dbstate":{},"f:extra":{},"f:leader":{},"f:renew-time":{},"f:ttl":{}},"f:labels":{".":{},"f:app.kubernetes.io/instance":{},"f:app.kubernetes.io/managed-by":{},"f:apps.kubeblocks.io/component-name":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"6e19661b-3bd0-4cf2-a1b5-6c6dd211fa60\"}":{}}}}}]}}},"Members":[{"Index":"","Name":"asmysql-zileht-mysql-0","Role":"","PodIP":"10.96.1.34","DBPort":"3306","SyncerPort":"3601","UID":"35e1edf8-4425-4565-ab7d-b4bb6aaf8a9e","ComponentName":"mysql","UseIP":false}],"Switchover":null,"Extra":null,"Resource":{"metadata":{"name":"asmysql-zileht","namespace":"default","uid":"6e19661b-3bd0-4cf2-a1b5-6c6dd211fa60","resourceVersion":"51966","generation":1,"creationTimestamp":"2024-06-28T06:45:54Z","labels":{"clusterdefinition.kubeblocks.io/name":"mysql","clusterversion.kubeblocks.io/name":"mysql-5.7.44"},"annotations":{"kubeblocks.io/ops-request":"[{\"name\":\"asmysql-zileht-restart-pv5tc\",\"type\":\"Restart\"}]","kubeblocks.io/reconcile":"2024-06-28T07:38:40.850367374Z"},"finalizers":["cluster.kubeblocks.io/finalizer"],"managedFields":[{"manager":"kbcli83","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2024-06-28T06:45:54Z","fieldsType":"FieldsV1","fieldsV1":{"f:spec":{".":{},"f:affinity":{".":{},"f:podAntiAffinity":{},"f:tenancy":{}},"f:clusterDefinitionRef":{},"f:clusterVersionRef":{},"f:componentSpecs":{},"f:monitor":{},"f:resources":{".":{},"f:cpu":{},"f:memory":{}},"f:storage":{".":{},"f:size":{}},"f:terminationPolicy":{}}}},{"manager":"manager","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2024-06-28T07:32:30Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:clusterDefGeneration":{},"f:components":{".":{},"f:mysql":{".":{},"f:phase":{},"f:podsReady":{},"f:podsReadyTime":{}}},"f:conditions":{},"f:observedGeneration":{},"f:phase":{}}},"subresource":"status"},{"manager":"manager","operation":"Update","apiVersion":"apps.kubeblocks.io/v1alpha1","time":"2024-06-28T07:38:40Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubeblocks.io/ops-request":{},"f:kubeblocks.io/reconcile":{}},"f:finalizers":{".":{},"v:\"cluster.kubeblocks.io/finalizer\"":{}},"f:labels":{".":{},"f:clusterdefinition.kubeblocks.io/name":{},"f:clusterversion.kubeblocks.io/name":{}}}}}]},"spec":{"clusterDefinitionRef":"mysql","clusterVersionRef":"mysql-5.7.44","terminationPolicy":"Delete","componentSpecs":[{"name":"mysql","componentDefRef":"mysql","enabledLogs":["auditlog","error","slow"],"replicas":1,"resources":{"limits":{"cpu":"100m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"512Mi"}},"volumeClaimTemplates":[{"name":"data","spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"10Gi"}}}}],"switchPolicy":{"type":"Noop"},"serviceAccountName":"kb-asmysql-zileht"}],"affinity":{"podAntiAffinity":"Preferred","tenancy":"SharedNode"},"resources":{"cpu":"0","memory":"0"},"storage":{"size":"0"},"monitor":{}},"status":{"observedGeneration":1,"phase":"Updating","components":{"mysql":{"phase":"Updating","podsReady":false,"podsReadyTime":"2024-06-28T06:47:45Z"}},"clusterDefGeneration":2,"conditions":[{"type":"ProvisioningStarted","status":"True","observedGeneration":1,"lastTransitionTime":"2024-06-28T06:45:54Z","reason":"PreCheckSucceed","message":"The operator has started the provisioning of Cluster: asmysql-zileht"},{"type":"ApplyResources","status":"True","observedGeneration":1,"lastTransitionTime":"2024-06-28T06:45:54Z","reason":"ApplyResourcesSucceed","message":"Successfully applied for resources"},{"type":"ReplicasReady","status":"False","lastTransitionTime":"2024-06-28T07:32:30Z","reason":"ReplicasNotReady","message":"pods are not ready in Components: [mysql], refer to related component message in Cluster.status.components"},{"type":"Ready","status":"False","lastTransitionTime":"2024-06-28T07:32:30Z","reason":"ComponentsNotReady","message":"pods are unavailable in Components: [mysql], refer to related component message in Cluster.status.components"}]}}}} 2024-06-28T07:40:31Z INFO HA try to initialize cluster 2024-06-28T07:40:31Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:40:36Z INFO HA try to initialize cluster 2024-06-28T07:40:36Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:40:41Z INFO HA try to initialize cluster 2024-06-28T07:40:41Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:40:46Z INFO HA try to initialize cluster 2024-06-28T07:40:46Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:40:51Z INFO HA try to initialize cluster 2024-06-28T07:40:51Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:40:56Z INFO HA try to initialize cluster 2024-06-28T07:40:56Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:41:01Z INFO HA try to initialize cluster 2024-06-28T07:41:01Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:41:06Z INFO HA try to initialize cluster 2024-06-28T07:41:06Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:41:11Z INFO HA try to initialize cluster 2024-06-28T07:41:11Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:41:16Z INFO HA try to initialize cluster 2024-06-28T07:41:16Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:41:21Z INFO HA try to initialize cluster 2024-06-28T07:41:21Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:41:26Z INFO HA try to initialize cluster 2024-06-28T07:41:26Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:41:31Z INFO HA try to initialize cluster 2024-06-28T07:41:31Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:41:36Z INFO HA try to initialize cluster 2024-06-28T07:41:36Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:41:41Z INFO HA try to initialize cluster 2024-06-28T07:41:41Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:41:46Z INFO HA try to initialize cluster 2024-06-28T07:41:46Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:41:51Z INFO HA try to initialize cluster 2024-06-28T07:41:51Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:41:56Z INFO HA try to initialize cluster 2024-06-28T07:41:56Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:42:01Z INFO HA try to initialize cluster 2024-06-28T07:42:01Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:42:06Z INFO HA try to initialize cluster 2024-06-28T07:42:06Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:42:11Z INFO HA try to initialize cluster 2024-06-28T07:42:11Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:42:16Z INFO HA try to initialize cluster 2024-06-28T07:42:16Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:42:21Z INFO HA try to initialize cluster 2024-06-28T07:42:21Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:42:26Z INFO HA try to initialize cluster 2024-06-28T07:42:26Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:42:31Z INFO HA try to initialize cluster 2024-06-28T07:42:31Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:42:36Z INFO HA try to initialize cluster 2024-06-28T07:42:36Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:42:41Z INFO HA try to initialize cluster 2024-06-28T07:42:41Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:42:46Z INFO HA try to initialize cluster 2024-06-28T07:42:46Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263 2024-06-28T07:42:51Z INFO HA try to initialize cluster 2024-06-28T07:42:51Z ERROR HA Cluster initialize failed {"error": "Ha Configmap has been created", "errorVerbose": "Ha Configmap has been created\ngithub.com/apecloud/syncer/dcs.(KubernetesStore).CreateHaConfig\n\t/src/dcs/k8s.go:482\ngithub.com/apecloud/syncer/highavailability.(Ha).initializeClusterIdentifier\n\t/src/highavailability/ha.go:452\ngithub.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig\n\t/src/highavailability/ha.go:422\ngithub.com/apecloud/syncer/highavailability.(Ha).Start\n\t/src/highavailability/ha.go:263\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"} github.com/apecloud/syncer/highavailability.(Ha).startDBAndCreateHaConfig /src/highavailability/ha.go:424 github.com/apecloud/syncer/highavailability.(Ha).Start /src/highavailability/ha.go:263

➜ ~ k logs kubeblocks-5fcff48447-h4tsk -n kb-system >kb.txt Defaulted container "manager" out of: manager, tools (init), datascript (init)


[kb.txt](https://github.com/user-attachments/files/16026128/kb.txt)
JashBook commented 4 months ago
  1. upgrade kb from 0.8.3 to 0.9.0-beta.41
  2. kbcli addon upgrade mysql --version 0.9.1
  3. kbcli cluster restart -n ns-cluster asm-cluster --auto-approve --force
  4. see error 
    kbcli cluster list-instances asm-cluster -n ns-cluster
NAME                  NAMESPACE    CLUSTER       COMPONENT   STATUS    ROLE        ACCESSMODE   AZ           CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)           STORAGE     NODE                                                      CREATED-TIME                 
asm-cluster-mysql-0   ns-cluster   asm-cluster   mysql       Running   secondary   <none>       us-west-2a   200m / 200m          644245094400m / 644245094400m   data:10Gi   ip-172-31-25-96.us-west-2.compute.internal/172.31.25.96   Jun 28,2024 15:31 UTC+0800   
asm-cluster-mysql-1   ns-cluster   asm-cluster   mysql       Running   secondary   <none>       us-west-2a   200m / 200m          644245094400m / 644245094400m   data:10Gi   ip-172-31-31-81.us-west-2.compute.internal/172.31.31.81   Jun 28,2024 15:32 UTC+0800   
asm-cluster-mysql-2   ns-cluster   asm-cluster   mysql       Running   <none>      <none>       us-west-2a   200m / 200m          644245094400m / 644245094400m   data:10Gi   ip-172-31-27-34.us-west-2.compute.internal/172.31.27.34   Jun 28,2024 16:48 UTC+0800
  5. see mysql error log 
    kubectl exec -it asm-cluster-mysql-2  ns-cluster  -n ns-cluster  bash
cat /var/lib/mysql/log/mysqld-error.log
2024-06-28T07:09:38.691164Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-06-28T07:09:38.691190Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_exec_mode instead.
2024-06-28T07:09:38.691469Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.33) initializing of server in progress as process 84
2024-06-28T07:09:38.702328Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-06-28T07:09:41.613886Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-06-28T07:09:46.397720Z 0 [Warning] [MY-013501] [Server] Ignoring --plugin-load[_add] list as the server is running with --initialize(-insecure).
2024-06-28T07:09:56.394117Z 0 [Warning] [MY-000067] [Server] unknown variable 'loose_audit_log_handler=FILE'.
2024-06-28T07:09:56.394143Z 0 [Warning] [MY-000067] [Server] unknown variable 'loose_audit_log_file=/var/lib/mysql/auditlog/audit.log'.
2024-06-28T07:09:56.394152Z 0 [Warning] [MY-000067] [Server] unknown variable 'loose_audit_log_buffer_size=1Mb'.
2024-06-28T07:09:56.394160Z 0 [Warning] [MY-000067] [Server] unknown variable 'loose_audit_log_policy=ALL'.
2024-06-28T07:09:56.394168Z 0 [Warning] [MY-000067] [Server] unknown variable 'loose_audit_log_strategy=ASYNCHRONOUS'.
2024-06-28T07:09:56.394175Z 0 [Warning] [MY-000067] [Server] unknown variable 'loose_audit_log_rotate_on_size=10485760'.
2024-06-28T07:09:56.394182Z 0 [Warning] [MY-000067] [Server] unknown variable 'loose_audit_log_rotations=5'.
2024-06-28T07:09:56.395378Z 5 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2024-06-28T07:10:19.201542Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-06-28T07:10:19.201597Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_exec_mode instead.
2024-06-28T07:10:19.201994Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33) starting as process 125
2024-06-28T07:10:19.392986Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-06-28T07:10:21.592559Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-06-28T07:10:25.304301Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-06-28T07:10:25.304347Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-06-28T07:10:25.309014Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2024-06-28T07:10:25.604498Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  MySQL Community Server - GPL.
2024-06-28T07:10:26.791412Z 8 [System] [MY-013172] [Server] Received SHUTDOWN from user root. Shutting down mysqld (Version: 8.0.33).
2024-06-28T07:10:28.197980Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33)  MySQL Community Server - GPL.
2024-06-28T07:10:29.103968Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-06-28T07:10:29.103997Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_exec_mode instead.
2024-06-28T07:10:29.104373Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33) starting as process 1
2024-06-28T07:10:29.198151Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-06-28T07:10:31.192499Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-06-28T07:10:33.400802Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-06-28T07:10:33.400858Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-06-28T07:10:33.406023Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2024-06-28T07:10:33.696156Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
2024-06-28T07:10:45.686426Z 7 [Warning] [MY-011153] [Repl] Timeout waiting for reply of binlog (file: mysql-bin.000003, pos: 1256), semi-sync up to file , position 0.
2024-06-28T07:18:19.943591Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-06-28T07:18:19.943620Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_exec_mode instead.
2024-06-28T07:18:19.943997Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33) starting as process 1
2024-06-28T07:18:19.954501Z 0 [Warning] [MY-010075] [Server] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 98bc512c-351e-11ef-9acc-7acda00b7321.
2024-06-28T07:18:20.040831Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-06-28T07:18:22.774393Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-06-28T07:18:25.253791Z 0 [System] [MY-010229] [Server] Starting XA crash recovery...
2024-06-28T07:18:25.443299Z 0 [System] [MY-010232] [Server] XA crash recovery finished.
2024-06-28T07:18:25.755238Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-06-28T07:18:25.755296Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-06-28T07:18:25.759888Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2024-06-28T07:18:26.139892Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
2024-06-28T07:18:34.151543Z 7 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL '' executed'. Previous state source_host='', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''. New state source_host='asm-cluster-mysql-0.asm-cluster-mysql-headless', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''.
2024-06-28T07:18:34.168392Z 8 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2024-06-28T07:18:34.251736Z 8 [System] [MY-014002] [Repl] Replica receiver thread for channel '': connected to source 'root@asm-cluster-mysql-0.asm-cluster-mysql-headless:3306' with server_uuid=620ade22-351d-11ef-9f3c-bee71569868b, server_id=1. Starting GTID-based replication.
2024-06-28T07:18:53.477199Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.33).
2024-06-28T07:18:55.743222Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33)  MySQL Community Server - GPL.
2024-06-28T07:20:02.848610Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-06-28T07:20:02.848649Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_exec_mode instead.
2024-06-28T07:20:02.849150Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33) starting as process 1
2024-06-28T07:20:02.955982Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-06-28T07:20:05.888921Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-06-28T07:20:08.558210Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-06-28T07:20:08.558251Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-06-28T07:20:08.563802Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2024-06-28T07:20:08.869120Z 0 [Warning] [MY-010539] [Repl] Recovery from source pos 44847 and file mysql-bin.000003 for channel ''. Previous relay log pos and relay log file had been set to 17007, ./relay-bin.000002 respectively.
2024-06-28T07:20:08.945207Z 5 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2024-06-28T07:20:09.038695Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
2024-06-28T07:20:09.041718Z 5 [System] [MY-014002] [Repl] Replica receiver thread for channel '': connected to source 'root@asm-cluster-mysql-0.asm-cluster-mysql-headless:3306' with server_uuid=620ade22-351d-11ef-9f3c-bee71569868b, server_id=1. Starting GTID-based replication.
2024-06-28T07:20:40.744699Z 5 [System] [MY-010558] [Repl] Replica for channel '': received end packet from server due to dump thread being killed on source. Dump threads are killed for example during source shutdown, explicitly by a user, or when the source receives a binlog send request from a duplicate server UUID <98bc512c-351e-11ef-9acc-7acda00b7321> : Error 
2024-06-28T07:20:40.744880Z 5 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2024-06-28T07:20:40.749260Z 5 [ERROR] [MY-010584] [Repl] Replica I/O for channel '': Error reconnecting to source 'root@asm-cluster-mysql-0.asm-cluster-mysql-headless:3306'. This was attempt 1/86400, with a delay of 60 seconds between attempts. Message: Can't connect to MySQL server on 'asm-cluster-mysql-0.asm-cluster-mysql-headless:3306' (111), Error_code: MY-002003
2024-06-28T07:22:02.991723Z 0 [ERROR] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-001158 - Got an error reading communication packets
2024-06-28T07:22:05.288026Z 0 [ERROR] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-001158 - Got an error reading communication packets
2024-06-28T07:22:10.294859Z 13 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL '' executed'. Previous state source_host='asm-cluster-mysql-0.asm-cluster-mysql-headless', source_port= 3306, source_log_file='mysql-bin.000004', source_log_pos= 2816, source_bind=''. New state source_host='asm-cluster-mysql-0.asm-cluster-mysql-headless', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''.
2024-06-28T07:22:10.324178Z 27 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2024-06-28T07:22:10.438361Z 27 [System] [MY-014002] [Repl] Replica receiver thread for channel '': connected to source 'root@asm-cluster-mysql-0.asm-cluster-mysql-headless:3306' with server_uuid=620ade22-351d-11ef-9f3c-bee71569868b, server_id=1. Starting GTID-based replication.
2024-06-28T07:28:47.123012Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.33).
2024-06-28T07:28:49.541524Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 22  user: 'root'.
2024-06-28T07:28:50.543359Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33)  MySQL Community Server - GPL.
2024-06-28T07:29:33.941177Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-06-28T07:29:33.941215Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_exec_mode instead.
2024-06-28T07:29:33.941826Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33) starting as process 1
2024-06-28T07:29:33.957559Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-06-28T07:29:35.086488Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-06-28T07:29:36.321999Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-06-28T07:29:36.322041Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-06-28T07:29:36.339674Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2024-06-28T07:29:36.459842Z 0 [Warning] [MY-010539] [Repl] Recovery from source pos 1870139 and file mysql-bin.000005 for channel ''. Previous relay log pos and relay log file had been set to 1864104, ./relay-bin.000002 respectively.
2024-06-28T07:29:36.461927Z 5 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2024-06-28T07:29:36.541130Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
2024-06-28T07:29:36.543154Z 5 [System] [MY-014002] [Repl] Replica receiver thread for channel '': connected to source 'root@asm-cluster-mysql-0.asm-cluster-mysql-headless:3306' with server_uuid=620ade22-351d-11ef-9f3c-bee71569868b, server_id=1. Starting GTID-based replication.
2024-06-28T07:32:05.741447Z 0 [ERROR] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-001158 - Got an error reading communication packets
2024-06-28T07:33:01.734151Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.33).
2024-06-28T07:33:01.744733Z 13 [Warning] [MY-011145] [Repl] SEMISYNC: Forced shutdown. Some updates might not be replicated.
2024-06-28T07:33:11.752996Z 0 [Warning] [MY-010909] [Server] /usr/sbin/mysqld: Forcing close of thread 30  user: 'root'.
2024-06-28T07:33:12.540639Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33)  MySQL Community Server - GPL.
2024-06-28T07:33:49.861137Z 0 [Warning] [MY-011068] [Server] The syntax '--skip-host-cache' is deprecated and will be removed in a future release. Please use SET GLOBAL host_cache_size=0 instead.
2024-06-28T07:33:49.861325Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_exec_mode instead.
2024-06-28T07:33:49.861592Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.33) starting as process 1
2024-06-28T07:33:49.954580Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2024-06-28T07:33:50.912118Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2024-06-28T07:33:51.960609Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2024-06-28T07:33:51.960648Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2024-06-28T07:33:51.964891Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2024-06-28T07:33:52.071291Z 0 [Warning] [MY-010539] [Repl] Recovery from source pos 1870347 and file mysql-bin.000005 for channel ''. Previous relay log pos and relay log file had been set to 628, ./relay-bin.000004 respectively.
2024-06-28T07:33:52.142831Z 5 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2024-06-28T07:33:52.154791Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.
2024-06-28T07:33:52.247167Z 5 [System] [MY-014002] [Repl] Replica receiver thread for channel '': connected to source 'root@asm-cluster-mysql-0.asm-cluster-mysql-headless:3306' with server_uuid=620ade22-351d-11ef-9f3c-bee71569868b, server_id=1. Starting GTID-based replication.
2024-06-28T07:33:56.589565Z 13 [System] [MY-010597] [Repl] 'CHANGE REPLICATION SOURCE TO FOR CHANNEL '' executed'. Previous state source_host='asm-cluster-mysql-0.asm-cluster-mysql-headless', source_port= 3306, source_log_file='mysql-bin.000007', source_log_pos= 10286, source_bind=''. New state source_host='asm-cluster-mysql-1.asm-cluster-mysql-headless', source_port= 3306, source_log_file='', source_log_pos= 4, source_bind=''.
2024-06-28T07:33:56.611682Z 14 [Warning] [MY-010897] [Repl] Storing MySQL user name or password information in the connection metadata repository is not secure and is therefore not recommended. Please consider using the USER and PASSWORD connection options for START REPLICA; see the 'START REPLICA Syntax' in the MySQL Manual for more information.
2024-06-28T07:33:56.659648Z 14 [System] [MY-014002] [Repl] Replica receiver thread for channel '': connected to source 'root@asm-cluster-mysql-1.asm-cluster-mysql-headless:3306' with server_uuid=6ee52023-351e-11ef-9af5-7a93eea8d05b, server_id=2. Starting GTID-based replication.
2024-06-28T08:47:54.605731Z 0 [System] [MY-013172] [Server] Received SHUTDOWN from user <via user signal>. Shutting down mysqld (Version: 8.0.33).
2024-06-28T08:47:56.743936Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33)  MySQL Community Server - GPL.