search

tabular-io / iceberg-kafka-connect

Apache License 2.0
177 stars 32 forks source link

Is there any option to ingest sink after specific offset? #137

Closed okayhooni closed 9 months ago

okayhooni commented 9 months ago

In the case of too large Kafka topic, I want to ingest Iceberg table sink only after specific offset. (Then, I will insert previous data to iceberg table with a separate batch job.)

But, the consumer group auto-created by this sink connector, always try to ingest from first record on the topic in retention period.

Is there any option or configuration to ingest sink after specific consumer offset?

(I think it is useful if there is some option like reset-offset-to-latest when initiating sink connector.)

okayhooni commented 9 months ago

self-answer:

sorry.. I found the option I expected, on the kafka connect itself. "consumer.override.auto.offset.reset": "latest"