pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
Apache License 2.0
417 stars 272 forks source link

[Limitation] TiCDC can't not handle DDL as `alter table xx add column xx datetime default current_timestamp` properly #11368

Open asddongmen opened 2 days ago

asddongmen commented 2 days ago

What did you do?

Reproduction Steps

  1. Create a table t1

    create table t1 (id int primary key);
    
  2. Create a changefeed to replicate this table

  3. Insert some data into t1

  4. Pause changefeed (the purpose is to manually create synchronization delay), assuming the current time is time1.

  5. Add a new column to t1

    alter table t1 add column `updatetime` datetime default current_timestamp;
    
  6. Resume the changefeed and wait for the changefeed to replicate the above DDL downstream. Assume the current time is time2.

    When comparing the upstream and downstream data, it will be found that the values of the newly added column updatetime upstream and downstream inconsistent. The difference in this column upstream and downstream is time2 - time1.

Reason: This is because cdc only synchronizes the DDL itself when synchronizing DDLs that will cause destructive changes like add column, and does not synchronize the row changes caused by the DDL. The reason for doing this is to reduce synchronization latency.

Currently, these inconsistent data can only be manually fixed.

What did you expect to see?

No response

What did you see instead?

As described above.

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

All version.
lance6716 commented 1 day ago

if it's caused by different value of current_timestamp in upsteam and downstream, it should be able to fix like https://github.com/pingcap/tiflow/pull/7036

asddongmen commented 20 hours ago

@lance6716 Thank you very much for your useful insights!