search

10xfuturetechnologies / kafka-connect-iceberg

Kafka Connector for Iceberg tables
Apache License 2.0
16 stars 5 forks source link

Re-writing the same data for each run even though there is no new event in kafka #33

Open haripriyarhp opened 1 year ago

haripriyarhp commented 1 year ago

Hi, Is there somewhere where you make note of the kafka offsets that has been processed. Because I just sent 5 events to Kafka but it looks like the connector keeps on writing the 5 messages again and again in S3. Even though querying the iceberg table returns only the 5 records, the s3 objects keep increasing even though there are no new messages. Am I missing some parameter? Because 403 objects for just 5 events is a bit too much and if the connector is running, then the objects keep increasing even if there are no new kafka events

image