search

uber / marmaray

Generic Data Ingestion & Dispersal Library for Hadoop
https://eng.uber.com/marmaray-hadoop-ingestion-open-source/
Other
478 stars 111 forks source link

mysql connector #6

Open saravsars opened 6 years ago

saravsars commented 6 years ago

Trying to use marmaray for mysql to hive (hoodie format) pipeline but not able to find mysql/jdbc source in code. Any plans to add it in the future?

esmioley commented 6 years ago

Hi @saravsars, thanks for your interest. We do plan on adding integration with mysql in specific and probably jdbc in generic, but do not currently have an ETA for such a feature. If you're interested in helping us add that support, it would be greatly appreciated.

Just as an FYI, at Uber we have a team that is producing avro-encoded kafka messages based off of the MySQL binlog, and so we managed to ingest them via the kafka source to produce the table.