search

pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
Apache License 2.0
424 stars 283 forks source link

"RowChangedEvent fallback row, ignore it" when injecting CDC and Kfaka consumer network partition #11237

Closed fubinzh closed 2 months ago

fubinzh commented 3 months ago

What did you do?

  1. Topo: Upstream TiDB --CDC-- Kafka -- Consumer -- MySQL
  2. create Kafka open protocol changefeed: /cdc cli changefeed create "--server=127.0.0.1:8301" "--sink-uri=kafka://downstream-kafka.cdc-kafka-controller-tps-7508966-1-706:9092/cdc_kafka_network_partition?replication-factor=2&max-message-bytes=1048576&protocol=open-protocol" "--changefeed-id=kafka-open-protocol-task-1
  3. Run sysbench prepare
  4. Run sysbench workload and at the same time inject CDC owner to Kafka controller network partition.
  5. Run kafka consumer to consumer the kafka msg to MySQL
  6. Do data consistency check between TiDB and MySQL

What did you expect to see?

Kafka consumer should not panic, and kafka msg consumed correctly and data consistency check passed.

What did you see instead?

Consumer shoult not panic, and data consistency check passed betwen TiDB and MySQL

[writer.go:362] ["RowChangedEvent fallback row, ignore it"] [commitTs=450190641590173785] [watermark=450190655496913022] [partition=2] [row="{\"StartTs\":0,\"CommitTs\":450190641590173785,\"RowID\":0,\"PhysicalTableID\":0,

Versions of the cluster

CDC : 350a2fb4596b489af648982082e60ea5735178ff Consumer: c70aaf2590c0efcf4c8cce58d52ed88e7c20d287

fubinzh commented 3 months ago

/severity major

3AceShowHand commented 3 months ago

This is caused by the message duplication, no message loss, this message can be ignored and does not affect eventual consistency.

We are going to investigate this issue further

3AceShowHand commented 2 months ago

This issue cannot be reproduced, close it. If reproduced, reopen this issue