ddl: fast reorg of index may cause duplicate entry error

pingcap / tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.

https://pingcap.com

Apache License 2.0

37.28k stars 5.85k forks source link

ddl: fast reorg of index may cause duplicate entry error #49233

Open AilinKid opened 11 months ago

AilinKid commented 11 months ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

2. What did you expect to see? (Required)

Look, we did add a unique index with ddl job id as 1131, while at 02.19 ddl owner has changed, and resume the checkpoint at same time, consequently leading duplicate entry error at 02.21

3. What did you see instead (Required)

no duplicate error (since user has checked with their data)

4. What is your TiDB version? (Required)

master, 7.1.1

lance6716 commented 6 months ago

Same type of root cause like https://github.com/pingcap/tidb/issues/46986

lance6716 commented 6 months ago

after https://github.com/pingcap/tidb/pull/52993 , there are still 2 cases can trigger the false alert of "duplicate entry"

non distributed execution framework: due to concurrency DDL worker and low watermark checkpoint, some data will be imported twice with different TS when DDL owner changes
distributed execution framework with local sort: there's no checkpoint manager so TS is allocated when needed and does not persist

cc @tangenta

kennedy8312 commented 6 months ago

Are there any pending tasks for this issue? Shall we close this one?

lance6716 commented 6 months ago

Are there any pending tasks for this issue? Shall we close this one?

It's still not totally fixed, I have listed 2 cases in above comments. It's hard to fix, maybe we can fix it in v8.3