search

radondb / radondb-mysql-kubernetes

Open Source，High Availability Cluster，based on MySQL
Apache License 2.0
352 stars 83 forks source link

Leader is read_only #27

Open andyli029 opened 3 years ago

andyli029 commented 3 years ago

Summary

mysql> show variables like '%read_only%';
+-----------------------+-------+
| Variable_name         | Value |
+-----------------------+-------+
| innodb_read_only      | OFF   |
| read_only             | ON    |
| super_read_only       | OFF   |
| transaction_read_only | OFF   |
| tx_read_only          | OFF   |
+-----------------------+-------+
runkecheng commented 3 years ago

The environment before the failure

master : test-krypton-2

slave : test-krypton-0 and test-krypton-1

mysql user/database : qingcloud/qingcloud

operate

Run the script on the master：

sysbench /usr/share/sysbench/oltp_read_write.lua  --mysql-port=31933 --mysql-db=qingcloud --mysql-user=qingcloud --mysql-password=Qing@123 --table_size=100000 --tables=4 --threads=128 --events=0 --report-interval=10 --time=600 --mysql-host=192.168.0.4  --table_size=100000 prepare/run

Kill the xenon container of test-krypton-0

kill 1

Then the slave node xenon container restarts. After the restart is successful, the master node is demoted to the slave node. About half a minute later, the election of the master ends, and the new master node is still test-krypton-2,but user qingcloud becomes read-only when connecting to the master node

Description：Not sure if it will switch to another master

log

the xenon log of master 

2021/04/12 06:10:14.202707 trace.go:37:      [ERROR]    LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:4, E:2].send.heartbeat.get.rsp[N:test-krypton-0.test-krypton.default.svc.cluster.local:8801, V:0, E:0].error[ErrorInvalidRequest]
 2021/04/12 06:10:16.116558 trace.go:32:         [WARNING]  LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:4, E:2].heartbeat.acks.granted[2].less.than.members[3].for.the.first.time
 2021/04/12 06:10:16.116603 trace.go:32:         [WARNING]  LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:4, E:2].do.updateViewID[FROM:4 TO:6]
 2021/04/12 06:10:18.005117 api.go:280:      [INFO]     mysql.slave.status:&{ 0  false  false        }
 2021/04/12 06:10:18.005664 api.go:290:      [INFO]     mysql.master.status:&{mysql-bin.000014 13890118  true  true   04ca5655-62bf-42f4-80d3-572dd93832b4:1-1692,
227c185e-b737-4522-a8ca-5799c926fd18:1-458,
2ca566eb-565e-4564-8c75-4215b2adeae4:1-162675,
37870b92-ca10-4c70-b9cd-532830336418:1-104862,
5a92301f-0756-4fd9-9000-786799e77d85:1-1328,
5ac149ed-d597-41f2-bfc3-58fabdd5e65d:1-270043,
851bea4b-f8c0-4222-911b-ad7162372bdd:1-5,
ac3e0ae4-76ac-4a30-b404-1d4ed86e61a3:1-8176,
b51580ed-7817-44cf-aa06-eae55ec6a6c8:1-8,
cf108a7a-f809-4ac7-a45a-317502c399cc:1-249 0    }
 2021/04/12 06:10:18.116858 trace.go:32:         [WARNING]  LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:6, E:2].heartbeat.acks.granted[3].equals.members[3].again
 2021/04/12 06:10:23.004012 api.go:280:      [INFO]     mysql.slave.status:&{ 0  false  false        }
 2021/04/12 06:10:23.103157 api.go:290:      [INFO]     mysql.master.status:&{mysql-bin.000014 14630010  true  true   04ca5655-62bf-42f4-80d3-572dd93832b4:1-1692,
227c185e-b737-4522-a8ca-5799c926fd18:1-458,
2ca566eb-565e-4564-8c75-4215b2adeae4:1-162675,
37870b92-ca10-4c70-b9cd-532830336418:1-104862,
5a92301f-0756-4fd9-9000-786799e77d85:1-1328,
5ac149ed-d597-41f2-bfc3-58fabdd5e65d:1-270043,
851bea4b-f8c0-4222-911b-ad7162372bdd:1-5,
ac3e0ae4-76ac-4a30-b404-1d4ed86e61a3:1-8608,
b51580ed-7817-44cf-aa06-eae55ec6a6c8:1-8,
cf108a7a-f809-4ac7-a45a-317502c399cc:1-249 0    }
 2021/04/12 06:10:28.003700 api.go:280:      [INFO]     mysql.slave.status:&{ 0  false  false        }
 2021/04/12 06:10:28.108516 api.go:290:      [INFO]     mysql.master.status:&{mysql-bin.000014 15369902  true  true   04ca5655-62bf-42f4-80d3-572dd93832b4:1-1692,
227c185e-b737-4522-a8ca-5799c926fd18:1-458,
2ca566eb-565e-4564-8c75-4215b2adeae4:1-162675,
37870b92-ca10-4c70-b9cd-532830336418:1-104862,
5a92301f-0756-4fd9-9000-786799e77d85:1-1328,
5ac149ed-d597-41f2-bfc3-58fabdd5e65d:1-270043,
851bea4b-f8c0-4222-911b-ad7162372bdd:1-5,
ac3e0ae4-76ac-4a30-b404-1d4ed86e61a3:1-9019,
b51580ed-7817-44cf-aa06-eae55ec6a6c8:1-8,
cf108a7a-f809-4ac7-a45a-317502c399cc:1-249 0    }
 2021/04/12 06:10:28.119336 trace.go:37:         [ERROR]    LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:6, E:2].send.heartbeat.to.peer[test-krypton-0.test-krypton.default.svc.cluster.local:8801].new.client.error
 [dial tcp: lookup test-krypton-0.test-krypton.default.svc.cluster.local: no such host]
 2021/04/12 06:10:28.119387 trace.go:37:         [ERROR]    LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:6, E:2].send.heartbeat.get.rsp[N:, V:0, E:0].error[ErrorRpcCall]
 2021/04/12 06:10:30.118012 trace.go:32:         [WARNING]  LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:6, E:2].heartbeat.acks.granted[2].less.than.members[3].for.the.first.time
 2021/04/12 06:10:30.118070 trace.go:32:         [WARNING]  LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:6, E:2].do.updateViewID[FROM:6 TO:8]
 2021/04/12 06:10:30.206376 trace.go:37:         [ERROR]    LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:8, E:2].send.heartbeat.to.peer[test-krypton-0.test-krypton.default.svc.cluster.local:8801].new.client.error
 [dial tcp: lookup test-krypton-0.test-krypton.default.svc.cluster.local: no such host]
 2021/04/12 06:10:30.206413 trace.go:37:         [ERROR]    LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:8, E:2].send.heartbeat.get.rsp[N:, V:0, E:0].error[ErrorRpcCall]
 2021/04/12 06:10:32.119549 trace.go:37:         [ERROR]    LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:8, E:2].send.heartbeat.to.peer[test-krypton-0.test-krypton.default.svc.cluster.local:8801].new.client.error
 [dial tcp: lookup test-krypton-0.test-krypton.default.svc.cluster.local: no such host]
 2021/04/12 06:10:32.119602 trace.go:37:         [ERROR]    LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:8, E:2].send.heartbeat.get.rsp[N:, V:0, E:0].error[ErrorRpcCall]
 2021/04/12 06:10:33.003533 api.go:280:      [INFO]     mysql.slave.status:&{ 0  false  false        }
 2021/04/12 06:10:33.103892 api.go:290:      [INFO]     mysql.master.status:&{mysql-bin.000014 16123370  true  true   04ca5655-62bf-42f4-80d3-572dd93832b4:1-1692,
227c185e-b737-4522-a8ca-5799c926fd18:1-458,
2ca566eb-565e-4564-8c75-4215b2adeae4:1-162675,
37870b92-ca10-4c70-b9cd-532830336418:1-104862,
5a92301f-0756-4fd9-9000-786799e77d85:1-1328,
5ac149ed-d597-41f2-bfc3-58fabdd5e65d:1-270043,
851bea4b-f8c0-4222-911b-ad7162372bdd:1-5,
ac3e0ae4-76ac-4a30-b404-1d4ed86e61a3:1-9489,
b51580ed-7817-44cf-aa06-eae55ec6a6c8:1-8,
cf108a7a-f809-4ac7-a45a-317502c399cc:1-249 0    }
 2021/04/12 06:10:34.119349 trace.go:37:         [ERROR]    LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:8, E:2].send.heartbeat.to.peer[test-krypton-0.test-krypton.default.svc.cluster.local:8801].new.client.error[dial tcp: lookup test-krypton-0.test-krypton.default.svc.cluster.local: no such host]
 2021/04/12 06:10:34.119383 trace.go:37:         [ERROR]    LEADER[ID:test-krypton-2.test-krypton.default.svc.cluster.local:8801, V:8, E:2].send.heartbeat.get.rsp[N:, V:0, E:0].error[ErrorRpcCall]
 2021/04/12 06:10:38.005485 api.go:280:      [INFO]     mysql.slave.status:&{ 0  false  false        }
 2021/04/12 06:10:38.110764 api.go:290:      [INFO]     mysql.master.status:&{mysql-bin.000014 16917566  true  true   04ca5655-62bf-42f4-80d3-572dd93832b4:1-1692,
227c185e-b737-4522-a8ca-5799c926fd18:1-458,
2ca566eb-565e-4564-8c75-4215b2adeae4:1-162675,
37870b92-ca10-4c70-b9cd-532830336418:1-104862,
5a92301f-0756-4fd9-9000-786799e77d85:1-1328,
5ac149ed-d597-41f2-bfc3-58fabdd5e65d:1-270043,
851bea4b-f8c0-4222-911b-ad7162372bdd:1-5,
ac3e0ae4-76ac-4a30-b404-1d4ed86e61a3:1-9926,
b51580ed-7817-44cf-aa06-eae55ec6a6c8:1-8,
cf108a7a-f809-4ac7-a45a-317502c399cc:1-249 0    }