search

pingcap / tidb

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
https://pingcap.com
Apache License 2.0
36.89k stars 5.81k forks source link

lightning import failed with error "Error 1062 (23000): Duplicate entry '-4' for key 'PK_GCOL_STORED10040.PRIMARY'" #45503

Open fubinzh opened 1 year ago

fubinzh commented 1 year ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. do a lightning import 
    ==lighting.toml==
lightning]
nlevel = 'debug'
[mydumper]
filter = ['coms.PK_*']
====
tidb-lightning "-d" "s3://nfs/lixia-tools-workload/comp1?access-key=xxx&secret-access-key=xxx&endpoint=http%3a%2f%2fminio.pingcap.net%3a9000&force-path-style=true" "-pd-urls" "upstream-pd.cdc-testbed-tps-1822014-1-706:2379" "-tidb-host" "upstream-tidb.cdc-testbed-tps-1822014-1-706" "-tidb-port" "4000" "-tidb-user" "root" "-tidb-password" "" "-backend" "tidb" "-sorted-kv-dir" "/tmp/sorted-kv-dir" "-c" "/tmp/tidb-lightning.toml

2. What did you expect to see? (Required)

3. What did you see instead (Required)

[2023/07/21 03:32:55.135 +00:00] [INFO] [import.go:1355] ["cancel periodic actions"] [do=false]
[2023/07/21 03:32:55.135 +00:00] [ERROR] [import.go:1580] ["restore all tables data failed"] [takeTime=39m29.924888289s] [error="[Lightning:
Restore:ErrRestoreTable]restore table `coms`.`PK_GCOL_STORED10040` failed: [`coms`.`PK_GCOL_STORED10040`] write rows exceed conflict thresho
ld: Error 1062 (23000): Duplicate entry '-4' for key 'PK_GCOL_STORED10040.PRIMARY'"]
[2023/07/21 03:32:55.135 +00:00] [ERROR] [import.go:505] ["run failed"] [step=4] [error="[Lightning:Restore:ErrRestoreTable]restore table `c
oms`.`PK_GCOL_STORED10040` failed: [`coms`.`PK_GCOL_STORED10040`] write rows exceed conflict threshold: Error 1062 (23000): Duplicate entry
'-4' for key 'PK_GCOL_STORED10040.PRIMARY'"]

4. What is your TiDB version? (Required)

[2023/07/21 02:27:55.843 +00:00] [INFO] [info.go:49] ["Welcome to TiDB-Lightning"] [release-version=v7.3.0-alpha] [git-hash=c6dcf2234594023674450f6b01d12cb49bd87ca7] [git-branch=heads/refs/tags/v7.3.0-alpha] [go-version=go1.20.6] [utc-build-time="2023-07-20 11:05:08"] [race-enabled=false]
lance6716 commented 1 year ago

The default behaviour of tidb backend has been changed. You can add 

[conflict]
strategy = "replace"

in configuration file