Shopify / ghostferry

The swiss army knife of live data migrations
https://shopify.github.io/ghostferry
MIT License
694 stars 65 forks source link

Added binlog events from MariaDB that can be skipped #297

Open HemeraOne opened 3 years ago

HemeraOne commented 3 years ago

As I'm not sure if this impacts any other MySQL versions I have to still perform some checks. Hope somebody can review this and perhaps say if there's any flaws resolves #296

shuhaowu commented 3 years ago

I'm a bit weary of changing the code like this upstream. We don't use MariaDB so can't perform any validations. IMO the best way to upstream this kind of code would be if the code written for MariaDB is only executed when some sort of configurable MariaDB mode is active. This is especially true this deep in the binlog streamer, as we don't want to accidentally introduce bugs that might lead to data corruption.

shuhaowu commented 3 years ago

Hope somebody can review this and perhaps say if there's any flaws

I'm not sure if I have enough context to review this code either. Skipping events like this may break interrupt and resume. Without a proper understanding (and good documentation) on why the ev.Pos is 0 and how MariaDB is different from MySQL, I don't really know how to review and merge this safely.

HemeraOne commented 3 years ago

Ok after looking again through the code I come to the conclusion I went in the complete wrong direction. For all events there's a binlog position except for 1 specific:

Each binlog file holds a new special Binlog Event, type 0xa4 (164), called START_ENCRIPTION event:

however documentation also states:

it’s written to disk but never sent to connected slave servers.

I retrieved it as following:

        case *replication.GenericEvent:
            fmt.Println("logPosc:", ev.Header.LogPos)
            fmt.Println("raw type:", ev.RawData[4])

logPosf: &{[153 64 231 96 164 60 0 0 0 40 0 0 0 0 0 0 0 128 0 1 1 0 0 0 131 225 116 62 161 1 56 114 104 248 22 224 93 202 121 243] 0xc00009a078 0xc000094090} raw type: 164

In this case we should be able to ignore binlog event with type 0xa4 (164)

HemeraOne commented 3 years ago

Created a bug report for MariaDB at https://jira.mariadb.org/browse/MDEV-26118 Updated the code to only specifically filter this binlog event

HemeraOne commented 3 years ago

They have reviewed the ticket and updated the documentation at https://mariadb.com/kb/en/start_encryption_event/ This means we can safely skip this event