search

apecloud / kubeblocks

KubeBlocks is an open-source control plane software that runs and manages databases, message queues and other stateful applications on K8s.
https://kubeblocks.io
GNU Affero General Public License v3.0
2.14k stars 176 forks source link

[BUG]mysql Slave_IO_Running is NO after Hscale or Promote #6495

Closed ahjing99 closed 9 months ago

ahjing99 commented 10 months ago 
➜  ~ kbcli version
Kubernetes: v1.27.7-gke.1121000
KubeBlocks: 0.8.1-beta.5
kbcli: 0.8.0

      `kbcli cluster list-instances asmysql-skqrmq --namespace default `

NAME                     NAMESPACE   CLUSTER          COMPONENT   STATUS    ROLE        ACCESSMODE   AZ              CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)   STORAGE    NODE                                                  CREATED-TIME
asmysql-skqrmq-mysql-0   default     asmysql-skqrmq   mysql       Running   secondary   <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:3Gi   gke-yjtest-default-pool-cd868a53-oilm/10.128.15.197   Jan 19,2024 14:46 UTC+0800
asmysql-skqrmq-mysql-1   default     asmysql-skqrmq   mysql       Running   secondary   <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:3Gi   gke-yjtest-default-pool-cd868a53-4i8k/10.128.15.194   Jan 19,2024 14:51 UTC+0800
asmysql-skqrmq-mysql-2   default     asmysql-skqrmq   mysql       Running   primary     <none>       us-central1-c   100m / 100m          512Mi / 512Mi           data:3Gi   gke-yjtest-default-pool-cd868a53-sjgv/10.128.0.46     Jan 19,2024 14:49 UTC+0800

      `kbcli cluster hscale asmysql-skqrmq --auto-approve --components mysql --replicas 2 --namespace default `

OpsRequest asmysql-skqrmq-horizontalscaling-pqtzt created successfully, you can view the progress:
    kbcli cluster describe-ops asmysql-skqrmq-horizontalscaling-pqtzt -n default

➜  ~ kbcli cluster connect asmysql-skqrmq
Connect to instance asmysql-skqrmq-mysql-0: out of asmysql-skqrmq-mysql-0(primary), asmysql-skqrmq-mysql-1(secondary)
Defaulted container "mysql" out of: mysql, metrics, kb-checkrole, config-manager
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 360
Server version: 5.7.44-log MySQL Community Server (GPL)

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SHOW SLAVE STATUS\G;
*************************** 1. row ***************************
               Slave_IO_State:
                  Master_Host: asmysql-skqrmq-mysql-2.asmysql-skqrmq-mysql-headless
                  Master_User: root
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000001
          Read_Master_Log_Pos: 12409
               Relay_Log_File: relay-bin.000002
                Relay_Log_Pos: 12622
        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: No
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 12409
              Relay_Log_Space: 12823
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 3
                  Master_UUID: f9a193b7-b696-11ee-9a7c-1a8b68796e1b
             Master_Info_File: /var/lib/mysql/data/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State:
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set: f9a193b7-b696-11ee-9a7c-1a8b68796e1b:1-45
            Executed_Gtid_Set: 759a9bc3-b696-11ee-92e3-f229b21c0fee:1-500,
75ce708c-b696-11ee-932d-faab7bc879ff:1-75,
f9a193b7-b696-11ee-9a7c-1a8b68796e1b:1-45
                Auto_Position: 1
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)

ERROR:
No query specified

mysql>

From secondary:

mysql> SHOW SLAVE STATUS\G;
*************************** 1. row ***************************
               Slave_IO_State:
                  Master_Host: asmysql-skqrmq-mysql-0.asmysql-skqrmq-mysql-headless
                  Master_User: root
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File:
          Read_Master_Log_Pos: 4
               Relay_Log_File: relay-bin.000001
                Relay_Log_Pos: 4
        Relay_Master_Log_File:
             Slave_IO_Running: No
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 0
              Relay_Log_Space: 154
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 1236
                Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'The slave is connecting using CHANGE MASTER TO MASTER_AUTO_POSITION = 1, but the master has purged binary logs containing GTIDs that the slave requires. Replicate the missing transactions from elsewhere, or provision a new slave from backup. Consider increasing the master's binary log expiration period. The GTID set sent by the slave is '75ce708c-b696-11ee-932d-faab7bc879ff:1-75', and the missing transactions are 'f9a193b7-b696-11ee-9a7c-1a8b68796e1b:1-45'.'
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
                  Master_UUID: 759a9bc3-b696-11ee-92e3-f229b21c0fee
             Master_Info_File: /var/lib/mysql/data/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp: 240119 07:13:32
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set: 75ce708c-b696-11ee-932d-faab7bc879ff:1-75
                Auto_Position: 1
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)

ERROR:
No query specified
ahjing99 commented 10 months ago

promote

  `kbcli cluster list-instances asmysql-aaahue --namespace default `

NAME                     NAMESPACE   CLUSTER          COMPONENT   STATUS    ROLE        ACCESSMODE   AZ              CPU(REQUEST/LIMIT)   MEMORY(REQUEST/LIMIT)           STORAGE    NODE                                                  CREATED-TIME
asmysql-aaahue-mysql-0   default     asmysql-aaahue   mysql       Running   secondary   <none>       us-central1-c   200m / 200m          644245094400m / 644245094400m   data:6Gi   gke-yjtest-default-pool-cd868a53-p9s5/10.128.15.196   Jan 19,2024 15:04 UTC+0800
asmysql-aaahue-mysql-1   default     asmysql-aaahue   mysql       Running   primary     <none>       us-central1-c   200m / 200m          644245094400m / 644245094400m   data:6Gi   gke-yjtest-default-pool-cd868a53-b4bu/10.128.15.195   Jan 19,2024 15:04 UTC+0800

k exec -it asmysql-aaahue-mysql-1 -c mysql sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
sh-4.2# mysql -uroot -pvtnmgjx6
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 108
Server version: 5.7.44-log MySQL Community Server (GPL)

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>  SHOW SLAVE STATUS\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: asmysql-aaahue-mysql-0.asmysql-aaahue-mysql-headless
                  Master_User: root
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000005
          Read_Master_Log_Pos: 111306
               Relay_Log_File: relay-bin.000002
                Relay_Log_Pos: 111479
        Relay_Master_Log_File: mysql-bin.000005
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 111306
              Relay_Log_Space: 111680
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
                  Master_UUID: 0e997620-b698-11ee-9227-4afd6f8a6b59
             Master_Info_File: /var/lib/mysql/data/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set: 0e997620-b698-11ee-9227-4afd6f8a6b59:91-498
            Executed_Gtid_Set: 0e997620-b698-11ee-9227-4afd6f8a6b59:1-498,
10243581-b698-11ee-925e-d26e353158e3:1-47
                Auto_Position: 1
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)

ERROR:
No query specified

kbcli cluster connect asmysql-aaahue
Connect to instance asmysql-aaahue-mysql-0: out of asmysql-aaahue-mysql-0(primary), asmysql-aaahue-mysql-1(secondary)
Defaulted container "mysql" out of: mysql, metrics, kb-checkrole, config-manager
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 149
Server version: 5.7.44-log MySQL Community Server (GPL)

Copyright (c) 2000, 2023, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SHOW SLAVE STATUS\G;
*************************** 1. row ***************************
               Slave_IO_State:
                  Master_Host: asmysql-aaahue-mysql-1.asmysql-aaahue-mysql-headless
                  Master_User: root
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000006
          Read_Master_Log_Pos: 8364
               Relay_Log_File: relay-bin.000006
                Relay_Log_Pos: 8537
        Relay_Master_Log_File: mysql-bin.000006
             Slave_IO_Running: No
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 8364
              Relay_Log_Space: 8738
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 2
                  Master_UUID: 10243581-b698-11ee-925e-d26e353158e3
             Master_Info_File: /var/lib/mysql/data/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State:
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set: 10243581-b698-11ee-925e-d26e353158e3:18-47
            Executed_Gtid_Set: 0e997620-b698-11ee-9227-4afd6f8a6b59:1-486,
10243581-b698-11ee-925e-d26e353158e3:1-47
                Auto_Position: 1
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)

ERROR:
No query specified
wangyelei commented 9 months ago

fixed at https://github.com/apecloud/kubeblocks-addons/pull/253