Duplicate message when partition revoking or rebalancing

[doubleverify-measurement]

Hi, We see a lot of duplicates message exactly when Re-balancing occurs.

• When rebalancing partitions are revoked and redistributed
• Trim files occurring
• Duplicate message at this moment

do you've any idea how to troubleshoot this behavior ? it seems that instead of trimming files it make more files

We use

• confluent last version
• Secor deployed on kubernetes
• Run on Google cloud write files on GCS bucket

Partition Revoking events

Trimming file events

Number of files uploaded watch out the picks on exact time trim files occurs

Thanks for your help

[ambud]

Are you using transactions in Kafka? if not consumer will be replaying messages.