search

radondb / xenon

The MySQL Cluster Autopilot Management with GTID and Raft
GNU General Public License v3.0
693 stars 162 forks source link

xenon mysql rebuildme 报错 #171

Open zhangwx0794 opened 1 year ago

zhangwx0794 commented 1 year ago

环境信息： centos7.6 perconna mysql 5.7.39

xenon版本信息： 最新版 [mysql@Xenon-3 bin]$ ./xenoncli version xenoncli:[{Tag: Time:2022/11/22 08:04:45 Git: GoVersion:go1.17.12 Platform:linux amd64}]

xenon mysql集群状态： image

报错信息如下： image

[mysql@Xenon-3 bin]$ /boss/soft/xenon/bin/xenoncli mysql backup --to=/u01/backup 2022/12/08 15:55:14.144046 [WARNING] rebuildme.found.best.slave[192.168.100.102:8801].leader[192.168.100.101:8801] 2022/12/08 15:55:14.144107 [WARNING] S1-->found.the.best.backup.host[192.168.100.102:8801].... 2022/12/08 15:55:14.148955 [WARNING] S2-->rm.and.mkdir.backupdir[/u01/backup] 2022/12/08 15:55:14.148971 [WARNING] S3-->xtrabackup.begin.... 2022/12/08 15:55:14.149458 [WARNING] rebuildme.backup.req[&{From: BackupDir:/u01/backup SSHHost:192.168.100.103 SSHUser:mysql SSHPasswd:mysql SSHPort:22 IOPSLimits:100000 XtrabackupBinDir:/usr/bin}].from[192.168.100.102:8801] 2022/12/08 15:55:33.293809 [WARNING] S3-->xtrabackup.end.... 2022/12/08 15:55:33.293827 [WARNING] S4-->apply-log.begin.... 2022/12/08 15:55:46.412498 [WARNING] S4-->apply-log.end.... 2022/12/08 15:55:46.412509 [WARNING] completed OK! 2022/12/08 15:55:46.412512 [WARNING] backup.all.done.... [mysql@Xenon-3 bin]$ /boss/soft/xenon/bin/xenoncli mysql rebuildme --from=192.168.100.102:8801 2022/12/08 15:56:07.973953 [WARNING] =====prepare.to.rebuildme===== IMPORTANT: Please check that the backup run completes successfully. At the end of a successful backup run innobackupex prints "completed OK!".

2022/12/08 15:56:07.974236 [WARNING] S1-->check.raft.leader 2022/12/08 15:56:07.996902 [WARNING] S2-->prepare.rebuild.from[192.168.100.102:8801].... 2022/12/08 15:56:08.044707 [WARNING] S3-->check.bestone[192.168.100.102:8801].is.OK.... 2022/12/08 15:56:08.044725 [WARNING] S4-->set.learner 2022/12/08 15:56:08.045802 [WARNING] S5-->stop.monitor 2022/12/08 15:56:08.046748 [WARNING] S6-->kill.mysql 2022/12/08 15:56:08.110453 [WARNING] S7-->check.bestone[192.168.100.102:8801].is.OK.... 2022/12/08 15:56:08.227050 [WARNING] S8-->clear.datadir[/u01/backup] 2022/12/08 15:56:08.237190 [WARNING] S9-->xtrabackup.begin.... 2022/12/08 15:56:08.237788 [WARNING] rebuildme.backup.req[&{From: BackupDir:/u01/backup SSHHost:192.168.100.103 SSHUser:mysql SSHPasswd:mysql SSHPort:22 IOPSLimits:100000 XtrabackupBinDir:/usr/bin}].from[192.168.100.102:8801] 2022/12/08 15:56:22.364753 [WARNING] S9-->xtrabackup.end.... 2022/12/08 15:56:22.364771 [WARNING] S10-->apply-log.begin.... 2022/12/08 15:56:31.849770 [WARNING] S10-->apply-log.end.... 2022/12/08 15:56:31.849780 [WARNING] S11-->start.mysql.begin... 2022/12/08 15:56:31.850831 [WARNING] S11-->start.mysql.end... 2022/12/08 15:56:31.850843 [WARNING] S12-->wait.mysqld.running.begin.... 2022/12/08 15:56:34.878475 [WARNING] wait.mysqld.running... 2022/12/08 15:56:37.893701 [WARNING] wait.mysqld.running... 2022/12/08 15:56:37.906427 [WARNING] S12-->wait.mysqld.running.end.... 2022/12/08 15:56:37.906441 [WARNING] S13-->wait.mysql.working.begin.... 2022/12/08 15:56:40.917830 [WARNING] wait.mysql.working... 2022/12/08 15:56:40.918250 [WARNING] S13-->wait.mysql.working.end.... 2022/12/08 15:56:40.918259 [WARNING] S14-->stop.and.reset.slave.begin.... 2022/12/08 15:56:41.024755 [WARNING] S14-->stop.and.reset.slave.end.... 2022/12/08 15:56:41.024768 [WARNING] S15-->reset.master.begin.... 2022/12/08 15:56:41.028661 [WARNING] S15-->reset.master.end.... mysql.go:482: unexpected error: info.file.content.invalid[mysql_binlog.000002 3581 ]

2022/12/08 15:56:41.036228 [PANIC] info.file.content.invalid[mysql_binlog.000002 3581 ] panic: [PANIC] info.file.content.invalid[mysql_binlog.000002 3581 ]

goroutine 1 [running]: xbase/xlog.(Log).Panic(0xc0000e81d0, {0x7989c3, 0x7aa7b7}, {0xc0000d9ad0, 0xc0000d9ae0, 0x3}) /boss/soft/xenon/src/xbase/xlog/xlog.go:142 +0xc8 cli/cmd.ErrorOK({0x8130a0, 0xc000198650}) /boss/soft/xenon/src/cli/cmd/common.go:35 +0x1af cli/cmd.mysqlRebuildMeCommandFn(0xc0000fbd40, {0xc0000e8260, 0x1, 0x1}) /boss/soft/xenon/src/cli/cmd/mysql.go:482 +0x15df vendor/github.com/spf13/cobra.(Command).execute(0xc0000fbd40, {0xc0000e8240, 0x1, 0x1}) /boss/soft/xenon/src/vendor/github.com/spf13/cobra/command.go:603 +0x3a7 vendor/github.com/spf13/cobra.(Command).ExecuteC(0xa5c740) /boss/soft/xenon/src/vendor/github.com/spf13/cobra/command.go:689 +0x24b vendor/github.com/spf13/cobra.(Command).Execute(...) /boss/soft/xenon/src/vendor/github.com/spf13/cobra/command.go:648 main.main() /boss/soft/xenon/src/cli/cli.go:43 +0x25

caphash commented 1 year ago

环境信息： centos7.6 perconna mysql 5.7.39

xenon版本信息： 最新版 [mysql@Xenon-3 bin]$ ./xenoncli version xenoncli:[{Tag: Time:2022/11/22 08:04:45 Git: GoVersion:go1.17.12 Platform:linux amd64}]

xenon mysql集群状态： image

报错信息如下： image

[mysql@Xenon-3 bin]$ /boss/soft/xenon/bin/xenoncli mysql backup --to=/u01/backup 2022/12/08 15:55:14.144046 [WARNING] rebuildme.found.best.slave[192.168.100.102:8801].leader[192.168.100.101:8801] 2022/12/08 15:55:14.144107 [WARNING] S1-->found.the.best.backup.host[192.168.100.102:8801].... 2022/12/08 15:55:14.148955 [WARNING] S2-->rm.and.mkdir.backupdir[/u01/backup] 2022/12/08 15:55:14.148971 [WARNING] S3-->xtrabackup.begin.... 2022/12/08 15:55:14.149458 [WARNING] rebuildme.backup.req[&{From: BackupDir:/u01/backup SSHHost:192.168.100.103 SSHUser:mysql SSHPasswd:mysql SSHPort:22 IOPSLimits:100000 XtrabackupBinDir:/usr/bin}].from[192.168.100.102:8801] 2022/12/08 15:55:33.293809 [WARNING] S3-->xtrabackup.end.... 2022/12/08 15:55:33.293827 [WARNING] S4-->apply-log.begin.... 2022/12/08 15:55:46.412498 [WARNING] S4-->apply-log.end.... 2022/12/08 15:55:46.412509 [WARNING] completed OK! 2022/12/08 15:55:46.412512 [WARNING] backup.all.done.... [mysql@Xenon-3 bin]$ /boss/soft/xenon/bin/xenoncli mysql rebuildme --from=192.168.100.102:8801 2022/12/08 15:56:07.973953 [WARNING] =====prepare.to.rebuildme===== IMPORTANT: Please check that the backup run completes successfully. At the end of a successful backup run innobackupex prints "completed OK!".

2022/12/08 15:56:07.974236 [WARNING] S1-->check.raft.leader 2022/12/08 15:56:07.996902 [WARNING] S2-->prepare.rebuild.from[192.168.100.102:8801].... 2022/12/08 15:56:08.044707 [WARNING] S3-->check.bestone[192.168.100.102:8801].is.OK.... 2022/12/08 15:56:08.044725 [WARNING] S4-->set.learner 2022/12/08 15:56:08.045802 [WARNING] S5-->stop.monitor 2022/12/08 15:56:08.046748 [WARNING] S6-->kill.mysql 2022/12/08 15:56:08.110453 [WARNING] S7-->check.bestone[192.168.100.102:8801].is.OK.... 2022/12/08 15:56:08.227050 [WARNING] S8-->clear.datadir[/u01/backup] 2022/12/08 15:56:08.237190 [WARNING] S9-->xtrabackup.begin.... 2022/12/08 15:56:08.237788 [WARNING] rebuildme.backup.req[&{From: BackupDir:/u01/backup SSHHost:192.168.100.103 SSHUser:mysql SSHPasswd:mysql SSHPort:22 IOPSLimits:100000 XtrabackupBinDir:/usr/bin}].from[192.168.100.102:8801] 2022/12/08 15:56:22.364753 [WARNING] S9-->xtrabackup.end.... 2022/12/08 15:56:22.364771 [WARNING] S10-->apply-log.begin.... 2022/12/08 15:56:31.849770 [WARNING] S10-->apply-log.end.... 2022/12/08 15:56:31.849780 [WARNING] S11-->start.mysql.begin... 2022/12/08 15:56:31.850831 [WARNING] S11-->start.mysql.end... 2022/12/08 15:56:31.850843 [WARNING] S12-->wait.mysqld.running.begin.... 2022/12/08 15:56:34.878475 [WARNING] wait.mysqld.running... 2022/12/08 15:56:37.893701 [WARNING] wait.mysqld.running... 2022/12/08 15:56:37.906427 [WARNING] S12-->wait.mysqld.running.end.... 2022/12/08 15:56:37.906441 [WARNING] S13-->wait.mysql.working.begin.... 2022/12/08 15:56:40.917830 [WARNING] wait.mysql.working... 2022/12/08 15:56:40.918250 [WARNING] S13-->wait.mysql.working.end.... 2022/12/08 15:56:40.918259 [WARNING] S14-->stop.and.reset.slave.begin.... 2022/12/08 15:56:41.024755 [WARNING] S14-->stop.and.reset.slave.end.... 2022/12/08 15:56:41.024768 [WARNING] S15-->reset.master.begin.... 2022/12/08 15:56:41.028661 [WARNING] S15-->reset.master.end.... mysql.go:482: unexpected error: info.file.content.invalid[mysql_binlog.000002 3581 ]

2022/12/08 15:56:41.036228 [PANIC] info.file.content.invalid[mysql_binlog.000002 3581 ] panic: [PANIC] info.file.content.invalid[mysql_binlog.000002 3581 ]

goroutine 1 [running]: xbase/xlog.(Log).Panic(0xc0000e81d0, {0x7989c3, 0x7aa7b7}, {0xc0000d9ad0, 0xc0000d9ae0, 0x3}) /boss/soft/xenon/src/xbase/xlog/xlog.go:142 +0xc8 cli/cmd.ErrorOK({0x8130a0, 0xc000198650}) /boss/soft/xenon/src/cli/cmd/common.go:35 +0x1af cli/cmd.mysqlRebuildMeCommandFn(0xc0000fbd40, {0xc0000e8260, 0x1, 0x1}) /boss/soft/xenon/src/cli/cmd/mysql.go:482 +0x15df vendor/github.com/spf13/cobra.(Command).execute(0xc0000fbd40, {0xc0000e8240, 0x1, 0x1}) /boss/soft/xenon/src/vendor/github.com/spf13/cobra/command.go:603 +0x3a7 vendor/github.com/spf13/cobra.(Command).ExecuteC(0xa5c740) /boss/soft/xenon/src/vendor/github.com/spf13/cobra/command.go:689 +0x24b vendor/github.com/spf13/cobra.(Command).Execute(...) /boss/soft/xenon/src/vendor/github.com/spf13/cobra/command.go:648 main.main() /boss/soft/xenon/src/cli/cli.go:43 +0x25

应该是 MySQL 集群没有开启 GTID，所以rebuildme 时候获取不到 GTID 记录报错。 需要开启 GTID。