delta-io / kafka-delta-ingest

A highly efficient daemon for streaming data from Kafka into Delta Lake
Apache License 2.0
337 stars 71 forks source link

Add mode that terminates kdi when the end of a topic is reached #153

Closed florissmit10 closed 8 months ago

florissmit10 commented 9 months ago

When the "-e" or "--ends_at_latest_offsets" is added as an argument the application terminates when the end of the topic is reached. The end of the topic is determined by storing the highest offsets known on startup. When a partitions reaches this offset, the partition is unassigned from the consumer.

Fixes #75

Curious to hear any feedback!

florissmit10 commented 8 months ago

Questions I have, which may require more testing.

florissmit10 commented 8 months ago

@rtyler What are your thoughts on the questions above? What would be needed in your opinion to merge this PR?