I found that duplicated records occurred on the CDC sink with this Iceberg sink connector after using spot nodes and activating the node consolidation feature of Karpenter. Although it happens very rarely, when it does occur, it tends to happen consecutively. In a related issue inquiry, @bryanck informed me that in the Iceberg version of the connector, safeguard logic has been added to ensure that no more than one coordinator task is running simultaneously during the connector rebalancing process.
Commit Contents
cherry-pick this safeguard logic from Apache version to the Tabular version as well.
the safeguard logic is designed to prevent more than one coordinator task from running simultaneously during connector task rebalancing, such as in cases where spot instances are terminated.
Context
I found that duplicated records occurred on the CDC sink with this Iceberg sink connector after using spot nodes and activating the node consolidation feature of Karpenter. Although it happens very rarely, when it does occur, it tends to happen consecutively. In a related issue inquiry, @bryanck informed me that in the Iceberg version of the connector, safeguard logic has been added to ensure that no more than one coordinator task is running simultaneously during the connector rebalancing process.
Commit Contents
Related Links
cc/ @fqtab