search

github / gh-ost

GitHub's Online Schema-migration Tool for MySQL
MIT License
12.44k stars 1.26k forks source link

gh-ost doesn't mind if you purge binary logs.... #629

Open tomkrouper opened 6 years ago

tomkrouper commented 6 years ago

I'm not sure if this is a bug or gh-ost just being polite. However, I was running a gh-ost migration where disk space was at a premium, so I was periodically going through and purging binary logs along the replication chain so I wouldn't run out of disk space. Right after I pressed on purging the master binary logs, I had the realization that I had just broken the migration. I checked the output of gh-ost and saw that it was on binary log mysql-bin.004134 and I had just purged to mysql-bin.004182. Now, gh-ost, being the polite program it was, it realized it had reached the "end" of 4134, and continued to "read" 4135, which it couldn't find, so it moved on to the next binary and the next until it got to 4182 and continued running from there .No error message, just messages like:

2018-08-17 18:49:58 INFO rotate to next log name: mysql-bin.004181
2018-08-17 18:49:58 INFO rotate to next log name: mysql-bin.004181

If gh-ost shouldn't bail out in these cases, it should at least scream loudly that it could not read the file.

cc: https://github.com/github/hookshot/pull/930

shlomi-noach commented 6 years ago

Thank you. Labeling this as high priority since it can cause data corruption.

shlomi-noach commented 6 years ago

Related: https://github.com/github/gh-ost/issues/627

shlomi-noach commented 6 years ago

"roatate" message originates at https://github.com/github/gh-ost/blob/d2726c77f86cb65e25bce5c0ac5fe3fa0c997488/go/binlog/gomysql_reader.go#L148

which directly uses the go-mysql library, which seems to produce those bogus events. I'll look into.

shlomi-noach commented 6 years ago

See ongoing discussion on #627

shlomi-noach commented 6 years ago

In https://github.com/github/gh-ost/issues/627#issuecomment-414120771 it appears to me like there is no bug.

However, @tomkrouper reports gh-ost was on mysql-bin.004134 while he pruged binlogs up to mysql-bin.004182, which is a lot of binary logs and a lot of events, surely more than 10240 events. And yet gh-ost did not complain. My attempt to reproduce show that gh-ost does complain on missing binlog entries. So I'm not sure what's going on here.

shlomi-noach commented 6 years ago

@tomkrouper do you happen to have the gh-ost log to share in private?