github / gh-ost

GitHub's Online Schema-migration Tool for MySQL
MIT License
12.43k stars 1.26k forks source link

on-failure hook not triggering #685

Open alexisde opened 5 years ago

alexisde commented 5 years ago

I was doing a long-ish migration on a disk constrained server that had a limit on the number of binlog files available.

After some amount of lag the required binlog file was deleted and binlogstreamer.go failed because it couldn't find it.

The problem is that I have a on-failure hook configured that should have been triggered and it didn't, so the migration was interrupted and I did not get the notification to fix the problem and start over.

Copy: 31257000/336329667 9.3%; Applied: 40758; Backlog: 0/1000; Time: 1h52m15s(total), 1h52m5s(copy); streamer: mysql-bin.011888:4; State: throttled, lag=247.464956s; ETA: 18h13m56s
2018/11/29 11:56:17 binlogsyncer.go:79: [info] create BinlogSyncer with config {99999 mysql 10.0.200.156 3306 ghost   false false <nil>}
2018/11/29 11:56:17 binlogsyncer.go:246: [info] begin to sync binlog from position (mysql-bin.011888, 4)
2018/11/29 11:56:17 binlogsyncer.go:139: [info] register slave for master server 10.0.200.156:3306
2018/11/29 11:56:17 binlogstreamer.go:47: [error] close sync with err: ERROR 1236 (HY000): Could not find first log file name in binary log index file
Copy: 31257000/336329667 9.3%; Applied: 40758; Backlog: 0/1000; Time: 1h52m20s(total), 1h52m10s(copy); streamer: mysql-bin.011888:4; State: throttled, lag=251.564998s; ETA: 18h14m45s
2018-11-29 11:56:22 FATAL 61 successive failures in streamer reconnect at coordinates mysql-bin.011888:4
shlomi-noach commented 5 years ago

Confirmed.