lightning: collect remote duplicate keys failed due to unexpected Delete key

[Chad20N13]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

[tikv-importer] backend = "local" incremental-import = true duplicate-resolution = 'remove' sorted-kv-dir = "/data5/sort_kv"

2. What did you expect to see? (Required)

import normally

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

Tidb: 5.2.3

lightning ： 5.4.0

[sleepymole]

It seems there are some other write operations during lightning importing. For local backend, it is not allowed to do any write operations through SQL on the target cluster, especially on the target table. Otherwise, data corruption could happen.

[Chad20N13]

No other write behavior

[sleepymole]

@Chad20N13 Does the target table contain data before importing?

[sleepymole]

TiDB Lightning duplicate resolution can detect duplicate data only within the data source. This feature cannot handle conflict with existing data before running TiDB Lightning.

See https://docs.pingcap.com/tidb/stable/tidb-lightning-error-resolution#duplicate-resolution-in-local-backend-mode.

[Chad20N13]

The upstream data source of the customer has duplicate primary keys or duplicate unique values. Now the requirement is: the downstream target table is the same table, divided into multiple imports, and the repeated conflicts are recorded during the import process and the duplicate values are deleted.

[sleepymole]

Currently, lightning doesn't support this scenario. It can be changed to a feature request. cc @sunzhaoyang

[Chad20N13]

@kennytm cccccccc, PTAL... thanks