search

go-mysql-org / go-mysql

a powerful mysql toolset with Go
MIT License
4.58k stars 976 forks source link

Canal module does not detect row updates in transactions after upgrading MariaDB to 11.4.1 #893

Closed mefcorvi closed 3 months ago

mefcorvi commented 3 months ago

Description

After upgrading our MariaDB database to version 11.4.1, the canal module of the go-mysql library stopped detecting row updates within transactions. This issue seems to be linked to the handling of fake rotate events that I added a few years ago. The problem stems from an incorrect assumption that events with LogPos=0 can be ignored. However, it appears that the new version of MariaDB does not specify LogPos for certain events:

Events written through transaction or statement cache have log_pos set
to 0 so that they can be copied directly to the binlog without having
to compute the real end_log_pos.

Relevant code in MariaDB: MariaDB log_event_server.cc - https://github.com/MariaDB/server/commit/b8f9f796ff8a1b0744cafde715c370db5a07adb7

Steps to Reproduce:

  1. Setup MariaDB 10.7.7.
  2. Configure the canal module to process binlog events.
  3. Execute a transaction that updates rows in the database.
  4. Check that the row updates are handled correctly.
  5. Upgrade MariaDB to version 11.4.1.
  6. Execute a transaction that updates rows in the database.
  7. Observe that the canal module does not detect the row updates.

Expected Behavior: The canal module should detect and process row updates within transactions, regardless of the LogPos value.

Actual Behavior: The canal module ignores events with LogPos=0, causing it to miss row updates within transactions.

Environment:

Additional Context

The issue seems to be related to a change in how MariaDB handles LogPos for events written through transaction or statement cache. To resolve this, the canal module needs to be updated to handle events with LogPos=0 appropriately.

Proposed Solution

Consider fake rotate events only for events of type ROTATE_EVENT with timestamp = 0. This aligns with the comment in the MySQL codebase indicating that the timestamp is set to 0 to distinguish FAKE_ROTATE_EVENT from ROTATE_EVENT and conforms to the MariaDB documentation on fake rotate events.

Related

https://github.com/go-mysql-org/go-mysql/issues/396 https://github.com/go-mysql-org/go-mysql/issues/406 https://github.com/go-mysql-org/go-mysql/issues/752

gaaf commented 2 months ago

With this fix row events are dispatched again, but with the position set to 0. For those that rely on the actual binlog position on every row event, use the MariaDB Server option binlog_legacy_event_pos = on. That will restore previous behavior.