databricks / iceberg-kafka-connect

Apache License 2.0
213 stars 47 forks source link

Question - Commit Coordinator Timing #282

Open evanzeller opened 2 months ago

evanzeller commented 2 months ago

In my setup I have a topic with data ready to go, when I start-up the connector writers start with PUTs before the commit interval passes and the coordinator emits a START_COMMIT. If all data is read and written before the first commit is started then nothing is ever committed unless fresh data comes on the topic.

Am I missing something, or shouldn't the coordinator emit a START_COMMIT immediately on startup?