yugabyte / debezium-connector-yugabytedb

A Debezium CDC connector for the YugabyteDB database
https://docs.yugabyte.com/stable/explore/change-data-capture/using-logical-replication/yugabytedb-connector/
Apache License 2.0
12 stars 8 forks source link

[DBZ] Use full partition ID for colocated tables #346

Closed vaibhav-yb closed 2 months ago

vaibhav-yb commented 3 months ago

Problem

Currently, the YBPartition class object returns a partition ID by the method getId() which depends whether the table is in colocated:

So all the offsets, keys, OpIds, etc. are stored against this ID.

Now since the service had special handling for snapshot for colocated tables, we treat every YBPartition as a colocated object in the snapshot phase - what that means is that we can get the snapshot GetChangesResponse for each colocated table individually. While we treat every YBPartition in streaming phase as non-colocated since we do not have a way to get the changes for each tablet separately.

This creates a problem at the management level that we have to take care of properly creating the YBPartition object otherwise we'll end up returning a wrong ID if we use YBPartition#getId method.

Solution

This PR removes the colocated property from the YBPartition class and introduces the following changes

Test plan

Ran existing tests for colocated tables as well as non colocated tables to ensure this PR breaks nothing.

vaibhav-yb commented 2 months ago

This PR is being closed in lieu of #347, #348 and #351