apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.57k stars 1.66k forks source link

Support reading from Kafka and writing to MySQL using SeaTunnel #7077

Open ragdollcb opened 3 days ago

ragdollcb commented 3 days ago

Search before asking

Description

I would like to request a feature that allows SeaTunnel to read data from Kafka and write it to MySQL. This feature would be incredibly useful for scenarios where data needs to be processed or transformed after being streamed to Kafka and then stored in a MySQL database for further analysis or use.

Usage Scenario

In my current project, I am using MySQL with Canal to write data changes into Kafka. I need to read this data from Kafka using SeaTunnel and then write it into MySQL. This would streamline the data flow and enable real-time data processing and storage, improving the efficiency of our data pipeline.

If implemented, this feature would enhance the versatility of SeaTunnel and make it a more powerful tool for data integration and ETL processes.

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

liunaijie commented 3 days ago

i think this feature already supported, kafka canal to mysql

ragdollcb commented 3 days ago

Could you please provide an example?

liunaijie commented 3 days ago

kafka source: https://seatunnel.apache.org/docs/2.3.5/connector-v2/source/kafka kafka canal json format: https://seatunnel.apache.org/docs/2.3.5/connector-v2/formats/canal-json/ mysql sink: https://seatunnel.apache.org/docs/2.3.5/connector-v2/sink/Mysql#cdcchange-data-capture-event

ragdollcb commented 3 days ago

thinks