conduitio-labs / conduit-connector-mysql

Conduit connector for MySQL
1 stars 1 forks source link

Feature: Support reading DDL commands from the binlog to allow for full db replication #34

Open nickchomey opened 3 months ago

nickchomey commented 3 months ago

Feature description

I would like to build an asynchronously-replicated/eventually-consistent multi-master cluster via Conduit + NATS (Galera has strong consistency). It seems to be a common use case with debezium (or maxwell's daemon) + kafka, but there's plenty of reasons to prefer Conduit + NATS (golang, simplicity, already using conduit and nats for other related things, and more).

But this requires that conduit read DDL commands from the binlog, in addition to DML.

In this other (now closed) issue #30, it was mentioned:

First, regarding DDLs. As you mention, debezium and maxwells include DDL statements as a query. That makes sense, as DDL statements are actually emitted by the DB in the binlog as queries. The library we are using also emits a replication.QueryEvent (which contains the field Query) in the OnDDL handler method. But OpenCDC records don't carry DDL data right now, they may however in the future, so that's a conversation for another time.

I hope that this can be given consideration!

lovromazgon commented 2 months ago

I opened a discussion in Conduit about detecting schema changes, I'd be interested to hear your take on this topic. https://github.com/ConduitIO/conduit/discussions/1762

nickchomey commented 2 months ago

Thanks for the consideration! I'll add my two cents there