MariaDb binlog compression

osheroff / mysql-binlog-connector-java

MySQL Binary Log connector

638 stars 160 forks source link

MariaDb binlog compression #8

Open akhawatrahTW opened 4 years ago

akhawatrahTW commented 4 years ago

Hi,

Trying mysql-binlog-connector 0.21.0 against mariadb. Whenever binlog compression is enabled, it throws java.io.IOException: Unknown event type.

Is binlog compression supported by mysql-binlog-connector ?

Following is how one of the events look (retrieved using mysqlbinlog):

Event{ header=EventHeaderV4{ timestamp=1592003319000, eventType=QUERY, serverId=5, headerLength=19, dataLength=280, nextPosition=753402908, flags=8 }, data=QueryEventData{ threadId=0, executionTime=0, errorCode=0, database='', sql='# Dummy event replacing event type 160 that slave cannot handle. ' } }

gunnarmorling commented 4 years ago

Ha, just came here to report the same issue :)

@osheroff, what's your general stance on MariaDB specifics? This comes up regularly in the Debezium community, and we'd like to support this going forward. We could spend some cycles on this, altough I'm not quite sure yet, whether this is something that can be done in general or not. I.e. some exploration of the details around binlog compression will be needed.

osheroff commented 4 years ago

I'd love to support mariaDB too, though not having a big mariadb production install to test against makes me nervous. The issues come up w/ Maxwell as well from time to time, and I've never personally found the time to invest. I guess I've just been waiting for that someone with the big mariaDB install to come help develop it.

bluebike commented 4 years ago

I have access to (big/real) MariaDB instances and would want to get it supported.

We haven't use binlog compression, but other features that has problems. Maybe we should have one issue for all MariaDB related things.

Binlog event checksums way understood by MariaDB (to get Debezium working this had to put this off)
MariaDB own event types (GTID_EVENT,GTID_LIST_EVENT) When using with replication protocol MariaDB doesn't send these by default, but when reading binlog file, library just fails. now.
MariaDB GTID
MariaDB 10.5 Has support for extended TABLE_MAP_EVENT (similar to MySQL 8.X)

Also getting MariaDB option part of automatic testing would nice...

There has been around few PRs related to MariaDB in shyiko/mysql-binlog-connector-java .

Support MariaDB GTID https://github.com/shyiko/mysql-binlog-connector-java/pull/50

Add mariadb event types https://github.com/shyiko/mysql-binlog-connector-java/pull/280

yahorsi commented 4 years ago

Same here, interested in MariaDb, using Maxwell and getting some errors (

Trigl commented 3 years ago

+1 Eager for MariaDB GTID