search

apache / seatunnel

SeaTunnel is a next-generation super high-performance, distributed, massive data integration tool.
https://seatunnel.apache.org/
Apache License 2.0
7.95k stars 1.8k forks source link

support data lake component as a source and sink will be useful #568

Closed lordk911 closed 4 years ago

lordk911 commented 4 years ago

as the doc : https://iceberg.apache.org/spark-structured-streaming/

is waterdrop have any plan to support data lake component

garyelephant commented 4 years ago

@lordk911 你好，data lake有计划支持，只是目前没有用户给我们提过相关需求，没有需要落地的公司。如果你这边需要尽快落地，我们可以提高data lake支持的优先级

lordk911 commented 4 years ago

我们这边在做数据湖组件的技术选型吧，不过目前考虑到没有相应的ETL工具来支持，我看waterdrop是支持spark-structured-streaming的，应该能比较快的集成像iceberg这样的组件做sink吧。