search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.08k stars 170 forks source link

[BUG] primary/secondary mysql `Slave_SQL_Running: No` after bench and rebulid instances #7704

Closed JashBook closed 3 months ago

JashBook commented 3 months ago

Describe the bug

kbcli version
Kubernetes: v1.28.9-gke.1289000
KubeBlocks: 0.9.0-beta.44
kbcli: 0.9.0-beta.2

Last_SQL_Error: Error 'Can't create database 'benchtest'; database exists' on query. Default database: 'benchtest'. Query: 'create database benchtest'

To Reproduce Steps to reproduce the behavior:

  1. create cluster 
    kbcli cluster create  asmysql-uedaby --termination-policy=Halt --cluster-definition=mysql --enable-all-logs=false --cluster-version=mysql-5.7.44 --set cpu=100m,memory=0.5Gi,replicas=2,storage=10Gi  
    
kbcli cluster list-instances asmysql-uedaby --namespace default

NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME
asmysql-uedaby-mysql-0 default asmysql-uedaby mysql Running secondary us-central1-b 100m / 100m 512Mi / 512Mi data:10Gi gke-infracreate-gke-kbdata-e2-standar-aded0170-jlsj/10.10.0.99 Jul 02,2024 20:36 UTC+0800
asmysql-uedaby-mysql-1 default asmysql-uedaby mysql Running primary us-central1-b 100m / 100m 512Mi / 512Mi data:10Gi gke-infracreate-gke-kbdata-e2-standar-aded0170-jlsj/10.10.0.99 Jul 02,2024 20:35 UTC+0800


2. benchmark

apiVersion: v1 kind: Pod metadata: name: benchtest-asmysql-uedaby namespace: default spec: containers:

kubectl get backup NAME POLICY METHOD REPO STATUS TOTAL-SIZE DURATION CREATION-TIME COMPLETION-TIME EXPIRATION-TIME backup-default-asmysql-uedaby-20240702204308 asmysql-uedaby-mysql-backup-policy xtrabackup backuprepo-kbcli-test Completed 1017384 17s 2024-07-02T12:43:12Z 2024-07-02T12:43:29Z 

4. rebulid instances

apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: asmysql-uedaby-rebuildinstance- namespace: default spec: type: RebuildInstance clusterRef: asmysql-uedaby force: true rebuildFrom:

kbcli cluster list-ops asmysql-uedaby --status all --namespace default

NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME
ops_status:asmysql-uedaby-rebuildinstance-qljdb default RebuildInstance asmysql-uedaby mysql Succeed 1/1 Jul 02,2024 20:49 UTC+0800

kbcli cluster list-instances asmysql-uedaby --namespace default

NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME
asmysql-uedaby-mysql-0 default asmysql-uedaby mysql Running secondary us-central1-a 100m / 100m 512Mi / 512Mi data:10Gi gke-infracreate-gke-kbdata-e2-standar-765d90c7-krbm/10.10.0.17 Jul 02,2024 20:49 UTC+0800
asmysql-uedaby-mysql-1 default asmysql-uedaby mysql Running primary us-central1-b 100m / 100m 512Mi / 512Mi data:10Gi gke-infracreate-gke-kbdata-e2-standar-aded0170-jlsj/10.10.0.99 Jul 02,2024 20:35 UTC+0800 

5. see error
connect secondary

kbcli cluster connect --namespace default -i asmysql-uedaby-mysql-0 Defaulted container "mysql" out of: mysql, metrics, lorry, config-manager, init-data (init), init-syncer (init), init-xtrabackup (init) mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 692 Server version: 5.7.44-log MySQL Community Server (GPL)

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SHOW SLAVE STATUS\G; 1. row Slave_IO_State: Waiting for master to send event Master_Host: asmysql-uedaby-mysql-1.asmysql-uedaby-mysql-headless Master_User: root Master_Port: 3306 Connect_Retry: 60 Master_Log_File: asmysql-uedaby-mysql-1-bin.000003 Read_Master_Log_Pos: 1326215 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 465 Relay_Master_Log_File: asmysql-uedaby-mysql-1-bin.000003 Slave_IO_Running: Yes Slave_SQL_Running: No Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 1007 Last_Error: Error 'Can't create database 'benchtest'; database exists' on query. Default database: 'benchtest'. Query: 'create database benchtest' Skip_Counter: 0 Exec_Master_Log_Pos: 26454 Relay_Log_Space: 1300427 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: NULL Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 1007 Last_SQL_Error: Error 'Can't create database 'benchtest'; database exists' on query. Default database: 'benchtest'. Query: 'create database benchtest' Replicate_Ignore_Server_Ids: Master_Server_Id: 2 Master_UUID: 8eb671e0-3868-11ef-978d-52efb96809c8 Master_Info_File: /var/lib/mysql/data/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: 240702 13:28:06 Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: 8eb671e0-3868-11ef-978d-52efb96809c8:77-1055 Executed_Gtid_Set: 71a188f1-3867-11ef-8087-6644b46324fa:1-1060, 8eb671e0-3868-11ef-978d-52efb96809c8:1-76 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: 1 row in set (0.00 sec)

ERROR: No query specified

mysql> SHOW MASTER STATUS\G; 1. row File: asmysql-uedaby-mysql-0-bin.000002 Position: 20695 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 71a188f1-3867-11ef-8087-6644b46324fa:1-1060, 8eb671e0-3868-11ef-978d-52efb96809c8:1-76 1 row in set (0.00 sec)

ERROR: No query specified

connect primary

kbcli cluster connect --namespace default -i asmysql-uedaby-mysql-1 Defaulted container "mysql" out of: mysql, metrics, lorry, config-manager, init-data (init), init-syncer (init), init-xtrabackup (init) mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 455 Server version: 5.7.44-log MySQL Community Server (GPL)

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SHOW SLAVE STATUS\G; 1. row Slave_IO_State: Master_Host: asmysql-uedaby-mysql-0.asmysql-uedaby-mysql-headless Master_User: root Master_Port: 3306 Connect_Retry: 60 Master_Log_File: asmysql-uedaby-mysql-0-bin.000003 Read_Master_Log_Pos: 288671 Relay_Log_File: relay-bin.000004 Relay_Log_Pos: 6184 Relay_Master_Log_File: asmysql-uedaby-mysql-0-bin.000003 Slave_IO_Running: No Slave_SQL_Running: No Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 288671 Relay_Log_Space: 6385 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: NULL Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 2003 Last_IO_Error: error reconnecting to master 'root@asmysql-uedaby-mysql-0.asmysql-uedaby-mysql-headless:3306' - retry-time: 60 retries: 1 Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: 71a188f1-3867-11ef-8087-6644b46324fa Master_Info_File: /var/lib/mysql/data/master.info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: 240702 12:36:22 Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: 71a188f1-3867-11ef-8087-6644b46324fa:1040-1060 Executed_Gtid_Set: 71a188f1-3867-11ef-8087-6644b46324fa:1-1060, 8eb671e0-3868-11ef-978d-52efb96809c8:1-1055 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: 1 row in set (0.00 sec)

ERROR: No query specified mysql> SHOW MASTER STATUS\G; 1. row File: asmysql-uedaby-mysql-1-bin.000003 Position: 1333568 Binlog_Do_DB: Binlog_Ignore_DB: Executed_Gtid_Set: 71a188f1-3867-11ef-8087-6644b46324fa:1-1060, 8eb671e0-3868-11ef-978d-52efb96809c8:1-1082 1 row in set (0.00 sec)

ERROR: No query specified



**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.
wangyelei commented 3 months ago

fixed at https://github.com/apecloud/kubeblocks-addons/pull/758