pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
Apache License 2.0
424 stars 283 forks source link

eventFilter incorrectly affects update events #11636

Open wk989898 opened 3 days ago

wk989898 commented 3 days ago

What did you do?

  1. create a changefeed with config
    [filter]
    rules = ['test.*']
    [[filter.event-filters]]
    matcher = ['test.t'] 
    ignore-event = ["delete"] # filter out delete event of table `test.t` 
  2. run sql
    create table t (a int primary key);
    insert into t values (1),(2);
    update t set a = 3 where a = 1;

What did you expect to see?

data should be equivalent

What did you see instead?

The following tables contains inconsistent data

+------------+---------+--------------------+----------------+---------+-----------+
|   TABLE    | RESULT  | STRUCTURE EQUALITY | DATA DIFF ROWS | UPCOUNT | DOWNCOUNT |
+------------+---------+--------------------+----------------+---------+-----------+
| `test`.`t` | succeed | true               | +0/-1          |       2 |         3 |
+------------+---------+--------------------+----------------+---------+-----------+

fix sql:

DELETE FROM `test`.`t` WHERE `a` = 1 LIMIT 1;

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

master
wk989898 commented 3 days ago

The reason is that update event will be splited into delete event and insert event, and delete event will be ignored.

fubinzh commented 3 days ago

/severity major