pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
Apache License 2.0
428 stars 286 forks source link

Event Reorder in The Same Region #5192

Closed nkorange closed 2 years ago

nkorange commented 2 years ago

Before asking a question, make sure you have

What is your question?

Is it expected that the events in the puller module is not ordered at region level?

I understand that the events are not globally ordered because the events are from different regions which belong to different Raft stores. But should the events in the same region be ordered when they are sent from TiKV?

I inserted some logs in the puller module and found they were not ordered in the same region.

liuzix commented 2 years ago

They are ordered as they are sent from TiKV, as long as there is no region merger or split. But the order might be different from what you might expect from the commitTs of the transactions.

The event order is different from the "logical order" as manifested by the commitTs, because TiKV sends events as they are applied to the underlying KV, which is the "physical order". To understand why the logical order and physical order are different, consider a scenario where there are two events committing (2pc), and they get from PD two timestamps 10 and 20. However, due to network delay, the 10 transaction was delayed for a longer period, causing it to reach TiKV later than the 20 event. In such a case, TiCDC would receive the 20 event first.

nkorange commented 2 years ago

@liuzix Thanks for the explanation.