Design a system which consumes columnar file notifications from HDFS/cloud storage and generates delta, hudi and iceberg metadata - Githubissues

apache / incubator-xtable

Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.

https://xtable.apache.org/

Apache License 2.0

919 stars 147 forks source link

Design a system which consumes columnar file notifications from HDFS/cloud storage and generates delta, hudi and iceberg metadata #554

Open vinishjail97 opened 1 month ago

vinishjail97 commented 1 month ago

Feature Request / Improvement

Sub-task of main feature request https://github.com/apache/incubator-xtable/issues/550

Look at Using cloud notifications queue in the approach mentioned below. It's only a high level approach, with many open ended items. We can do a design doc and finalize the approach through a discussion in dev mailing list. https://github.com/apache/incubator-xtable/issues/550#issuecomment-2386526797

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

vinishjail97 commented 1 month ago

Tagging people who were interested to contribute @sudharshanraja-db @purnachandergit in the design, let's discuss this in dev mailing list.

purnachandergit commented 1 month ago

I will be working on this