[xCluster] DDL Replication - Handle failovers when DDLs are being replicated

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.

https://www.yugabyte.com

Other

9.04k stars 1.08k forks source link

Open hulien22 opened 2 months ago

hulien22 commented 2 months ago

Jira Link: DB-12857 A few scenarios to handle here:

replicated DDL is currently running on the target
DDL is replicated, but safe time is not caught up to the time the DDL was run on the target
DDL is not yet replicated but data for the DDL has been replicated
batch of DDLs are being replicated (issue here is that safe time is still stuck at beginning of batch even if some of the DDLs succeed)

hari90 commented 1 month ago

Solution:

Pause replication on the group.
Wait for all Pollers to get the update and stop. 2.1. Dor DDL queue table we need to run all pending DDLs.
Flush tablet safe times to table
Compute namespace level safe time
PITR the database to the safe time 5.1 No need to PITR the metadata since that is logically at the xCluster safe time.

Also TODO: For DDL queue table send safe time in every batch (ignore the 250ms buffer).