databricks / iceberg-kafka-connect

Apache License 2.0
220 stars 49 forks source link

Question - Commit Coordinator Timing #282

Open evanzeller opened 3 months ago

evanzeller commented 3 months ago

In my setup I have a topic with data ready to go, when I start-up the connector writers start with PUTs before the commit interval passes and the coordinator emits a START_COMMIT. If all data is read and written before the first commit is started then nothing is ever committed unless fresh data comes on the topic.

Am I missing something, or shouldn't the coordinator emit a START_COMMIT immediately on startup?

fuzing commented 1 week ago

@evanzeller - Perhaps, is this your issue #308