search

radondb / xenon

The MySQL Cluster Autopilot Management with GTID and Raft
GNU General Public License v3.0
693 stars 162 forks source link

xenon rejoin xenon cluster is INVALID #144

Open deep-cds opened 3 years ago

deep-cds commented 3 years ago

你好 、 xenon 研发同学。 部署好的 xenon+mysql 集群已经运行了一段时间， 发现执行 /data/xenon/bin/xenoncli cluster mysql 命令会出现cluster.go:480: unexpected error: get.client.error[dial tcp 192.168.39.167:8801: i/o timeout],如下图所示。 image

但是过一会执行命令 ，出现 如下情况 image

后面我把这个 xenon 程序stop ，然后start 。发现xenon的状态是 INVALID

image

以下是日志 [16:02:13 root@k8s-node-01 xenon]# cat xenon.log xenon:[{Tag:v1.0.9-1-gc50a100 Time:2020/04/29 08:46:43 Git:c50a100 GoVersion:go1.14.2 Platform:linux amd64}] 2021/06/26 16:00:42.380592 [INFO] main: tag=[v1.0.9-1-gc50a100], git=[c50a100], goversion=[go1.14.2], builddate=[2020/04/29 08:46:43] 2021/06/26 16:00:42.380790 [WARNING] xenon.conf.raft:[&{MetaDatadir:raft.meta HeartbeatTimeout:1000 AdmitDefeatHtCnt:10 ElectionTimeout:3000 PurgeBinlogInterval:300000 SuperIDLE:false LeaderStartCommand: LeaderStopCommand: PurgeBinlogDisabled:false RequestTimeout:500 CandidateWaitFor2Nodes:60000}] 2021/06/26 16:00:42.380826 [WARNING] xenon.conf.mysql:[&{Admin:root Passwd:DOVYJ2gveVIirLzE Host:127.0.0.1 Port:3306 Basedir:/data/mysql8.0.22 Version:mysql57 DefaultsFile:/data/mysql8.0.22/config/my.cnf PingTimeout:1000 AdmitDefeatPingCnt:2 MasterSysVars: SlaveSysVars: ReplHost:192.168.39.167 ReplUser:repl ReplPasswd:replication}] 2021/06/26 16:00:42.380866 [WARNING] xenon.conf.mysqld:[&{SSHHost:192.168.39.167 SSHUser:root SSHPasswd:nnSnRp+xBKOFmSjF SSHPort:22 BackupDir:/data/mysql8.0.22/data XtrabackupBinDir:/usr/bin BackupIOPSLimits:100000 UseMemory:2GB Parallel:2 MysqldMonitorInterval:1000 Admin:root Passwd:DOVYJ2gveVIirLzE Host:127.0.0.1 Port:3306 Basedir:/data/mysql8.0.22 DefaultsFile:/data/mysql8.0.22/config/my.cnf}] 2021/06/26 16:00:42.381152 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].prepare.to.recovery.peers.from.[raft.meta].peers[[192.168.39.167:8801]].idlePeers[[]] 2021/06/26 16:00:42.381177 [INFO] server.prepare.setup.mysqlserver 2021/06/26 16:00:42.381186 [WARNING] mysqld.prepare.to.start... 2021/06/26 16:00:42.474056 [WARNING] mysqld.already.running... 2021/06/26 16:00:42.474078 [INFO] server.mysqlserver.setup.done 2021/06/26 16:00:42.474088 [INFO] server.mysql.wait.for.work[maxwait:60s] 2021/06/26 16:00:42.489500 [INFO] mysql.slave.status:&{mysql-bin.000004 57680655 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1958 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1958, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:00:42.489553 [INFO] server.mysql.gtid:{Master_Log_File:mysql-bin.000004 Read_Master_Log_Pos:57680655 Relay_Master_Log_File: Slave_IO_Running:true Slave_IO_Running_Str:Yes Slave_SQL_Running:true Slave_SQL_Running_Str:Yes Retrieved_GTID_Set:0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1958 Executed_GTID_Set:0cba804d-9845-11eb-b0db-fa163e9bad91:1-1958, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 Seconds_Behind_Master:0 Slave_SQL_Running_State:Slave has read all relay log; waiting for more updates Last_Error: Last_IO_Error: Last_SQL_Error:} 2021/06/26 16:00:42.489575 [INFO] server.mysql.set.to.READONLY 2021/06/26 16:00:42.490123 [INFO] server.mysql.start.slave 2021/06/26 16:00:42.490340 [INFO] server.mysql.check.replication.user... 2021/06/26 16:00:42.490686 [INFO] server.mysql.setup.done 2021/06/26 16:00:42.490701 [INFO] server.prepare.setup.RPC 2021/06/26 16:00:42.491009 [INFO] server.RPC.setup.done 2021/06/26 16:00:42.491036 [INFO] mysqld[/data/mysql8.0.22/config/my.cnf].monitor.start... 2021/06/26 16:00:42.491047 [INFO] mysql[127.0.0.1:3306].startping... 2021/06/26 16:00:42.491061 [INFO] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].raft.start... 2021/06/26 16:00:42.491175 [INFO] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].start.CheckBrainSplit 2021/06/26 16:00:42.491206 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].state.init 2021/06/26 16:00:42.491326 [WARNING] xrpc.Start.listening.on[192.168.39.167:8801] 2021/06/26 16:00:42.491337 [INFO] server.start.success... 2021/06/26 16:00:42.491343 [INFO] xenon.start.success... 2021/06/26 16:00:42.494777 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].leaderStopShellCommand[[-c ]].done 2021/06/26 16:00:42.494803 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].mysql.waitMysqlDoneAsync.prepare 2021/06/26 16:00:42.494842 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].state.machine.run 2021/06/26 16:00:42.495409 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].mysql.SetReadOnly.done 2021/06/26 16:00:42.495427 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].mysql.SetSlaveGlobalSysVar.done 2021/06/26 16:00:42.495436 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].prepareAsync.done 2021/06/26 16:00:42.501336 [INFO] mysql.slave.status:&{mysql-bin.000004 57680655 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1958 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1958, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:00:42.501398 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].init.my.gtid.is:{mysql-bin.000004 57680655 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1958 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1958, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:00:43.459841 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:43.991472 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:00:44.459370 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:45.459558 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:45.491951 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:00:46.460700 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:46.992236 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:00:47.462440 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:48.459498 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:48.492406 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:00:49.459956 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:49.992834 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:00:50.458966 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:51.463800 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:51.493678 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:00:52.459795 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:52.993987 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:00:53.460034 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:54.457774 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:54.494107 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:00:55.460327 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:55.994357 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:00:56.463465 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:57.462650 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:57.494515 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:00:58.462458 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:00:58.994784 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:00:59.460173 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:00.460270 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:00.494962 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:01.461057 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:01.995114 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:02.462786 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:03.463594 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:03.495539 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:04.463260 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:04.995809 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:05.462384 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:06.464705 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:06.495961 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:07.463703 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:07.996260 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:08.463654 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:09.464480 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:09.496800 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:10.460974 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:10.997015 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:11.463696 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:12.465038 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:12.497577 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:13.465436 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:13.997873 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:14.467418 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:15.463660 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:15.498224 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:16.464437 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:16.998408 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:17.464747 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:18.469714 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:18.499186 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:19.465306 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:19.999382 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:20.464325 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:21.463087 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:21.500581 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:22.464104 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:23.000820 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:23.466962 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:24.467790 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:24.501302 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:24.570185 [INFO] mysql.slave.status:&{mysql-bin.000004 57683413 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1962 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1962, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:01:25.466976 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:25.666384 [INFO] mysql.slave.status:&{mysql-bin.000004 57683413 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1962 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1962, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:01:26.001421 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:26.226163 [INFO] mysql.slave.status:&{mysql-bin.000004 57683413 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1962 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1962, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:01:26.466934 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:26.816453 [INFO] mysql.slave.status:&{mysql-bin.000004 57683413 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1962 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1962, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:01:27.349656 [INFO] mysql.slave.status:&{mysql-bin.000004 57683413 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1962 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1962, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:01:27.466789 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:27.501571 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:27.847386 [INFO] mysql.slave.status:&{mysql-bin.000004 57683413 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1962 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1962, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:01:28.467647 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:29.002103 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:29.467825 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:30.467635 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:30.503307 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:31.468236 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:32.003609 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:32.468363 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:33.469618 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:33.503849 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:34.472446 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:35.004252 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:35.470297 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:36.470724 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:36.505045 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:37.471026 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:38.005288 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:38.470004 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:39.470284 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:39.506212 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:40.469646 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:41.006448 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:41.471915 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:42.474189 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:42.506759 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:43.103250 [INFO] mysql.slave.status:&{mysql-bin.000004 57685766 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1964 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1964, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:01:43.472371 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:44.007034 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:44.474044 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:45.399258 [INFO] mysql.slave.status:&{mysql-bin.000004 57685766 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1964 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1964, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:01:45.472811 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:45.507228 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:46.338145 [INFO] mysql.slave.status:&{mysql-bin.000004 57685766 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1964 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1964, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:01:46.473672 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:47.007541 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:47.473403 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:47.968206 [INFO] mysql.slave.status:&{mysql-bin.000004 57685766 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1964 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1964, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:01:48.473309 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:48.507730 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:48.797818 [INFO] mysql.slave.status:&{mysql-bin.000004 57686187 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1965 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:01:49.475140 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:50.007921 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:50.008067 [INFO] mysql.slave.status:&{mysql-bin.000004 57686187 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1965 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:01:50.473337 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:50.832914 [INFO] mysql.slave.status:&{mysql-bin.000004 57686187 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1965 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:01:51.472657 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:51.508092 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:52.475658 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:53.008452 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:53.474967 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:54.473878 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:54.508610 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:55.474622 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:56.008813 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:56.475106 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:57.476667 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:57.509003 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:58.474316 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:01:59.009338 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:01:59.476117 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:02:00.474590 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:02:00.509589 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:02:01.473740 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:02:02.009840 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:02:02.476089 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:02:03.475916 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:02:03.509956 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:02:04.476939 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:02:05.010190 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:02:05.492110 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:02:06.477866 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].process.heartbeat.request.RetCode.not.OK:ErrorInvalidRequest 2021/06/26 16:02:06.510434 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:0].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:02:07.321738 [WARNING] server.rpc.node.add:&{From: Nodes:[192.168.39.162:8802 192.168.39.127:8803]} 2021/06/26 16:02:07.322225 [INFO] FOLLOWER[ID:192.168.39.167:8801, V:0, E:1].check.peers.json.file[raft.meta/peers.json].stat[name:peers.json, mode:-rwxr-xr-x, size:72, lastmodified:2021-06-26 16:02:07.321878393 +0800 CST] 2021/06/26 16:02:07.322280 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:1].add.peer[192.168.39.162:8802].to.peers[map[192.168.39.162:8802:0xc0001895c0]] 2021/06/26 16:02:07.329151 [INFO] FOLLOWER[ID:192.168.39.167:8801, V:0, E:2].check.peers.json.file[raft.meta/peers.json].stat[name:peers.json, mode:-rwxr-xr-x, size:94, lastmodified:2021-06-26 16:02:07.328878364 +0800 CST] 2021/06/26 16:02:07.329185 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:2].add.peer[192.168.39.127:8803].to.peers[map[192.168.39.127:8803:0xc000189710 192.168.39.162:8802:0xc0001895c0]] 2021/06/26 16:02:07.487994 [INFO] mysql.slave.status:&{mysql-bin.000004 57686187 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1965 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:02:07.488038 [WARNING] FOLLOWER[ID:192.168.39.167:8801, V:0, E:2].get.heartbeat.my.gtid.is:{mysql-bin.000004 57686187 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1965 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:02:07.546353 [INFO] mysql.get.local.uuid:[8b92218c-af9f-11eb-8207-fa163e0362cf] 2021/06/26 16:02:07.546920 [INFO] mysql.get.local.uuid:[8b92218c-af9f-11eb-8207-fa163e0362cf] 2021/06/26 16:02:07.547729 [WARNING] follower.gtid[&{mysql-bin.000004 57686187 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1829-1965 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }].bigger.than.remote[&{mysql-bin.000004 57686187 true true 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1965, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77658, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 }] 2021/06/26 16:02:07.547762 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].get.heartbeat.from[N:192.168.39.162:8802, V:105, E:2].change.mysql.master 2021/06/26 16:02:08.010688 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].ping.responses[1].is.less.than.half.maybe.brain.split 2021/06/26 16:02:08.013100 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].send.ping.to.peer[192.168.39.127:8803].client.call.ok.rsp[&{{0 0 FOLLOWER} { 0 false false } OK}] 2021/06/26 16:02:08.014062 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].send.ping.to.peer[192.168.39.162:8802].client.call.ok.rsp[&{{0 0 LEADER} { 0 false false } OK}] 2021/06/26 16:02:08.394047 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].get.heartbeat.change.to.the.new.master[192.168.39.162:8802].successed 2021/06/26 16:02:08.394078 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].get.heartbeat.from[N:192.168.39.162:8802, V:105, E:2].update.view 2021/06/26 16:02:08.394089 [WARNING] INVALID[ID:192.168.39.167:8801, V:0, E:2].do.updateViewID[FROM:0 TO:105] 2021/06/26 16:02:08.394458 [WARNING] INVALID[ID:192.168.39.167:8801, V:105, E:2].follower.state.machine.exit 2021/06/26 16:02:08.397859 [WARNING] INVALID[ID:192.168.39.167:8801, V:105, E:2].leaderStopShellCommand[[-c ]].done 2021/06/26 16:02:11.411235 [INFO] mysql.slave.status:&{mysql-bin.000004 57686633 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1966 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1966, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:02:13.316226 [INFO] mysql.slave.status:&{mysql-bin.000004 57686633 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1966 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1966, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:02:20.152371 [INFO] mysql.slave.status:&{mysql-bin.000004 57687054 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1966-1967 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1967, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:02:21.160871 [INFO] mysql.slave.status:&{mysql-bin.000004 57687054 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1966-1967 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1967, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates } 2021/06/26 16:02:21.740884 [INFO] mysql.slave.status:&{mysql-bin.000004 57687054 true Yes true Yes 0cba804d-9845-11eb-b0db-fa163e9bad91:1966-1967 0cba804d-9845-11eb-b0db-fa163e9bad91:1-1967, 8b92218c-af9f-11eb-8207-fa163e0362cf:1-77660, 933c3599-af9c-11eb-a5cb-fa163efebf78:1-33793, eb9aad49-9843-11eb-91a3-fa163e0362cf:1-6597 0 Slave has read all relay log; waiting for more updates }

请xenon 研发人员帮忙看一下。 谢谢

caphash commented 2 years ago

你好，问题可以分为两个部分解答：

  1. xenoncli 执行 timeout：

通过以下日志可以看到 xenon 配置 RequestTimeout 为 0.5 秒过小，在通常的网络条件下建议不小于 1 秒 并设置为 ElectionTimeout/AdmitDefeatHtCnt 的值。

2021/06/26 16:00:42.380790 [WARNING] xenon.conf.raft:[&{MetaDatadir:raft.meta HeartbeatTimeout:1000 AdmitDefeatHtCnt:10 ElectionTimeout:3000 PurgeBinlogInterval:300000 SuperIDLE:false LeaderStartCommand: LeaderStopCommand: PurgeBinlogDisabled:false RequestTimeout:500 CandidateWaitFor2Nodes:60000}] 2021/06/26 16:00:42.380826 [WARNING] xenon.conf.mysql:[&{Admin:root Passwd:DOVYJ2gveVIirLzE Host:127.0.0.1 Port:3306 Basedir:/data/mysql8.0.22 Version:mysql57 DefaultsFile:/data/mysql8.0.22/config/my.cnf PingTimeout:1000 AdmitDefeatPingCnt:2 MasterSysVars: SlaveSysVars: ReplHost:192.168.39.167 ReplUser:repl ReplPasswd:replication}] 2021/06/26 16:00:42.380866 [WARNING] xenon.conf.mysqld:[&{SSHHost:192.168.39.167 SSHUser:root SSHPasswd:nnSnRp+xBKOFmSjF SSHPort:22 BackupDir:/data/mysql8.0.22/data XtrabackupBinDir:/usr/bin BackupIOPSLimits:100000 UseMemory:2GB Parallel:2 MysqldMonitorInterval:1000 Admin:root Passwd:DOVYJ2gveVIirLzE Host:127.0.0.1 Port:3306 Basedir:/data/mysql8.0.22 DefaultsFile:/data/mysql8.0.22/config/my.cnf}]

  1. 主从发生过切换且原主 192.168.39.167 状态变为 INVALID：

这是由于主节点写数据时切换，leader 在降级后存在脏事务，当 xenon 检测到时自动将原主设置为 INVALID 状态，出现该情况后可以通过 ./xenoncli mysql rebuildme 进行数据重建，之后恢复为 FOLLOWER 状态。

另外，可以适当调大 HeartbeatTimeout、ElectionTimeout 等参数，并使用 xenon 最新版降低误切换可能性。

Lzw901 commented 5 months ago

[mysql@localhost /usr/local/xenon/bin]$ ./xenoncli mysql rebuildme --from=10.232.2.187:8801 2024/04/15 17:46:19.903706 [WARNING] =====prepare.to.rebuildme===== IMPORTANT: Please check that the backup run completes successfully. At the end of a successful backup run innobackupex prints "completed OK!".

2024/04/15 17:46:19.904072 [WARNING] S1-->check.raft.leader 2024/04/15 17:46:19.906042 [WARNING] S2-->prepare.rebuild.from[10.232.2.187:8801].... 2024/04/15 17:46:19.908912 [PANIC] I[10.232.2.188:8801].have.[2].local.transactions.more.than.maxAllowedLocalTrxCount[0].compared.to.from[10.232.2.187:8801].you.cant.rebuildme.sir panic: [PANIC] I[10.232.2.188:8801].have.[2].local.transactions.more.than.maxAllowedLocalTrxCount[0].compared.to.from[10.232.2.187:8801].you.cant.rebuildme.sir

goroutine 1 [running]: xbase/xlog.(Log).Panic(0xc0001862e0, {0x7b2085, 0x7fffa76da2da}, {0xc000121db0, 0xc000190750, 0x0}) /root/xenon-1.1.4/src/xbase/xlog/xlog.go:142 +0xc8 cli/cmd.mysqlRebuildMeCommandFn(0xc0001e6b40, {0xc000186370, 0x1, 0x1}) /root/xenon-1.1.4/src/cli/cmd/mysql.go:269 +0x438 vendor/github.com/spf13/cobra.(Command).execute(0xc0001e6b40, {0xc000186350, 0x1, 0x1}) /root/xenon-1.1.4/src/vendor/github.com/spf13/cobra/command.go:603 +0x3a7 vendor/github.com/spf13/cobra.(Command).ExecuteC(0xa5b720) /root/xenon-1.1.4/src/vendor/github.com/spf13/cobra/command.go:689 +0x24b vendor/github.com/spf13/cobra.(Command).Execute(...) /root/xenon-1.1.4/src/vendor/github.com/spf13/cobra/command.go:648 main.main() /root/xenon-1.1.4/src/cli/cli.go:43 +0x25

重建节点时报错这个是什么原因？