Connector: MYSQL [Source/Destination]

[simonl2002]

Resource name

MYSQL

Resource link

https://www.mysql.com/

Connector type

Source

Expected connector behavior

Connector should enable CDC stream of rows from mysql tables.

[nickchomey]

It isn't obvious because mysql isn't mentioned anywhere, but apparently the vitess connector already does this. I confirmed it with the team in the discord server.

[simonl2002]

@nickchomey we've decided to create a dedicated MYSQL connector. Vitess uses different terminology and it just feels easier to have dedicated connector.

[nickchomey]

Looks like work started on this a couple months ago. Just adding a link to it for others to discover. https://github.com/conduitio-labs/conduit-connector-mysql

[nadilas]

@nickchomey i understand the vitess connector is capable of handling a MySQL destination. Do you have an example setup? Can’t seem to connect from the connector to the MySQL instance (looking at the code suggests that it can only talk to a vitess vtable).

[nickchomey]

I never used it. Try the official mysql connector that they've been working on for a few months! It's linked above

[simonl2002]

@nadilas the MySQL connector is being worked on here: https://github.com/conduitio-labs/conduit-connector-mysql

[nadilas]

Thanks @simonl2002 @nickchomey, that connector does not have an implementation for a destination yet. I was hoping to get a temporary solution until the destination MySQL becomes ready.

[nickchomey]

There's a branch where they're working on the destination connector. Looks like many commits were done yesterday on getting tests working - that's probably a sign that it is very close to being merged.

Either wait for that, or you could try building that branch to see if it works for you (I only use the source connector). If it doesn't, you could report any bugs and that would help get it merged sooner.

[nadilas]

I was actually toying with the idea of doing the destination code ourselves, but couldn’t yet put an effort estimate to the idea to weigh the feasibility. Thanks for the tip, I’ll look into the branch. 👍

[hariso]

The destination PR was reviewed, so I think it should be merged relatively soon. The Conduit team will be focusing on the MySQL connector (and a few others) a bit more now that we're done with the 0.12 release.:)

[hariso]

I was actually toying with the idea of doing the destination code ourselves, but couldn’t yet put an effort estimate to the idea to weigh the feasibility. Thanks for the tip, I’ll look into the branch. 👍

Regarding doing the destination code yourselves: the Connector SDK should make it fairly simple to do so. The destination are simpler compared to source obviously and usually it's only batching that might require some additional though (Conduit makes batching possible, but doesn't and can't require connectors to do so). Just for the record, here are a few guidelines about writing connectors: https://github.com/ConduitIO/conduit/blob/4b66347eb89a2bb8cf641565f7602bf36bc33137/docs/writing-a-connector-guidelines.md.

[nickchomey]

I can confirm that destination connectors are fairly simple. I've made one for surrealdb.

But this one should be ready soon, so you might as well wait.

[nadilas]

We just tested it today and seems to work okay. We need to get some processors in to convert e.g sql server datetime string to MySQL format, but the basics seem to work. Will do some further testing tomorrow, thanks guys

[lovromazgon]

@nadilas you're welcome to join our Discord server (if you haven't already) and let us know how it goes or ask if something is unclear, we're happy to assist you 😉

[hariso]

FYI, the destination PR was merged.