search

github / gh-ost

GitHub's Online Schema-migration Tool for MySQL
MIT License
12.11k stars 1.23k forks source link

FATAL invalid sequence 8 != 1 #1415

Open jaco-louw opened 1 month ago

jaco-louw commented 1 month ago

Ubuntu 20.04.6 LTS (GNU/Linux 5.15.0-1058-aws x86_64) MySQL verison 8.0.36-28 Ghost Version - gh-ost-1.1.6-1.x86_64.rpm

ghost command:

gh-ost --max-load=Threads_running=25 --critical-load=Threads_running=1000 --chunk-size=1000 --max-lag-millis=1500 --user="myusername" --password="mypassword" --host=10.0.8.34 --database="test_db" --table="test_table" --verbose --alter="CHANGE COLUMN first_col VARCHAR(100) NULL DEFAULT NULL COLLATE 'utf8mb3_general_ci' AFTER second_col" --approve-renamed-columns --switch-to-rbr --allow-master-master --cut-over=default --exact-rowcount --concurrent-rowcount --default-retries=120 --initially-drop-ghost-table --initially-drop-socket-file --execute

error log:

2024-05-08 11:12:09 INFO starting gh-ost 1.1.6 2024-05-08 11:12:09 INFO Migrating test_db.test_table 2024-05-08 11:12:09 INFO Alter statement has column(s) renamed. gh-ost finds the following renames: map[first_col:VARCHAR(100)]; --approve-renamed-columns is given and so migration proceeds. 2024-05-08 11:12:09 INFO inspector connection validated on 10.0.8.34:3306 2024-05-08 11:12:09 INFO User has SUPER, REPLICATION SLAVE privileges, and has ALL privileges on test_db. 2024-05-08 11:12:09 INFO binary logs validated on 10.0.8.34:3306 2024-05-08 11:12:09 INFO Restarting replication on 10.0.8.34:3306 to make sure binlog settings apply to replication thread 2024-05-08 11:12:09 INFO Inspector initiated on ip-10-0-8-34:3306, version 8.0.36-28 2024-05-08 11:12:09 INFO Table found. Engine=InnoDB 2024-05-08 11:12:09 INFO Estimated number of rows via EXPLAIN: 256891 2024-05-08 11:12:09 INFO Recursively searching for replication master 2024-05-08 11:12:09 INFO Master found to be 10.0.9.34:3306 2024-05-08 11:12:09 INFO log_slave_updates validated on 10.0.8.34:3306 2024-05-08 11:12:09 INFO streamer connection validated on 10.0.8.34:3306 [2024/05/08 11:12:09] [info] binlogsyncer.go:148 create BinlogSyncer with config {99999 mysql 10.0.8.34 3306 myusername false false false UTC true 0 0s 0s 0 false false 0 } 2024-05-08 11:12:09 INFO Connecting binlog streamer at mysql-bin.322844:89692163 [2024/05/08 11:12:09] [info] binlogsyncer.go:374 begin to sync binlog from position (mysql-bin.322844, 89692163) 2024-05-08 11:12:09 INFO Tearing down inspector 2024-05-08 11:12:09 INFO Tearing down streamer 2024-05-08 11:12:09 FATAL invalid sequence 8 != 1 github.com/go-mysql-org/go-mysql/packet.(Conn).ReadPacketTo /home/meiji163/Documents/projects/gh-ost/vendor/github.com/go-mysql-org/go-mysql/packet/conn.go:140 github.com/go-mysql-org/go-mysql/packet.(Conn).ReadPacketReuseMem /home/meiji163/Documents/projects/gh-ost/vendor/github.com/go-mysql-org/go-mysql/packet/conn.go:98 github.com/go-mysql-org/go-mysql/client.(Conn).readResult /home/meiji163/Documents/projects/gh-ost/vendor/github.com/go-mysql-org/go-mysql/client/resp.go:218 github.com/go-mysql-org/go-mysql/client.(Conn).exec /home/meiji163/Documents/projects/gh-ost/vendor/github.com/go-mysql-org/go-mysql/client/conn.go:298 github.com/go-mysql-org/go-mysql/client.(Conn).Execute /home/meiji163/Documents/projects/gh-ost/vendor/github.com/go-mysql-org/go-mysql/client/conn.go:158 github.com/go-mysql-org/go-mysql/replication.(BinlogSyncer).registerSlave /home/meiji163/Documents/projects/gh-ost/vendor/github.com/go-mysql-org/go-mysql/replication/binlogsyncer.go:266 github.com/go-mysql-org/go-mysql/replication.(BinlogSyncer).prepare /home/meiji163/Documents/projects/gh-ost/vendor/github.com/go-mysql-org/go-mysql/replication/binlogsyncer.go:346 github.com/go-mysql-org/go-mysql/replication.(BinlogSyncer).prepareSyncPos /home/meiji163/Documents/projects/gh-ost/vendor/github.com/go-mysql-org/go-mysql/replication/binlogsyncer.go:617 github.com/go-mysql-org/go-mysql/replication.(BinlogSyncer).StartSync /home/meiji163/Documents/projects/gh-ost/vendor/github.com/go-mysql-org/go-mysql/replication/binlogsyncer.go:383 github.com/github/gh-ost/go/binlog.(GoMySQLReader).ConnectBinlogStreamer /home/meiji163/Documents/projects/gh-ost/go/binlog/gomysql_reader.go:64 github.com/github/gh-ost/go/logic.(EventsStreamer).initBinlogReader /home/meiji163/Documents/projects/gh-ost/go/logic/streamer.go:126 github.com/github/gh-ost/go/logic.(EventsStreamer).InitDBConnections /home/meiji163/Documents/projects/gh-ost/go/logic/streamer.go:116 github.com/github/gh-ost/go/logic.(Migrator).initiateStreaming /home/meiji163/Documents/projects/gh-ost/go/logic/migrator.go:1056 github.com/github/gh-ost/go/logic.(*Migrator).Migrate /home/meiji163/Documents/projects/gh-ost/go/logic/migrator.go:353 main.main /home/meiji163/Documents/projects/gh-ost/go/cmd/gh-ost/main.go:315 runtime.main /usr/lib/go-1.18/src/runtime/proc.go:250 runtime.goexit /usr/lib/go-1.18/src/runtime/asm_amd64.s:1571

We've been getting this error since having migrated from MySQL 5.7 to MySQL 8. What could be the issue? Any advice on what we can try to get gh-ost working again.