search

delta-io / kafka-delta-ingest

A highly efficient daemon for streaming data from Kafka into Delta Lake
Apache License 2.0
369 stars 80 forks source link

Write to multiple delta tables from a single topic #161

Open Kikkon opened 11 months ago

Kikkon commented 11 months ago

Referencing the design doc, I understand that if I want to write to multiple tables, I would need multiple processes and topics. This means I would need to manage the state of multiple topics & process.

However, since our company is relatively small, I think a single process would be sufficient for my needs.

So I would like to inquire about what the best practices are for this part? I think I could fork this repository and make some modifications, for example using a field in the topic as routing to write to different tables. But that feels a bit hack. I'm hoping for an approach more aligned with the current architecture. I'm happy to contribute back to the community if needed.

ThijsKoot commented 8 months ago

We're interested in this as well at our company.

mightyshazam commented 8 months ago

One way to keep it less hacky is to add the ability to filter messages after the transformation. This would still require you to run two processes, but it wouldn't require multiple topics. The idea of one consumer delivering multiple places is a layer of complexity that probably exists above this project.

leon-oosterwijk-akunacapital commented 4 months ago

I have this working on a private fork. trying to figure out how to share this.

Yevhenii-Kozlovskyi1 commented 3 months ago

Hi There! Has anyone had any luck implementing this approach, or encountered any blockers with this solution? I am interested in any implementation. Our team is thinking of changing our Delta Lake architecture due to scaling, performance, isolation, and delta-sharing things. One such approach is using a separate delta table per some input data characteristic which means our kafka-delta-ingest application should be able to recognize to which delta lake table the income message should be written based on some message properties.