lightning: support data KV checking of 'replace' mode for lightning post-import conflict detection

[lyzx2001]

What problem does this PR solve?

Issue Number: close #45774

Problem Summary:

Currently lightning only supports "remove" mode for post-import conflict detection, but many customers request lightning to support "replace" mode for lightning post-import conflict detection.

We would like to support "replace" mode for lightning post-import conflict detection: To resolve rows with conflict, instead of deleting all the rows that are engaged in conflict (the algorithm for remove), we delete some of the rows with conflict and reserve other rows that can be kept and not cause conflict anymore. Under this circumstance, we only delete the necessary rows to resolve conflicts, so that we can keep more original rows than remove mode as long as the conflicts are resolved.

The algorithms for index KV checking is contained in https://github.com/pingcap/tidb/pull/45926

This PR contains the algorithms for data KV checking.

What is changed and how it works?

Demo code for 'replace' mode of lightning post-import conflict detection:

https://github.com/lyzx2001/tidb-conflict-replace

Check List

Tests

• [x] Unit test
• [x] Integration test
• [ ] Manual test (add detailed scripts or steps below)
• [ ] No code

Side effects

• [ ] Performance regression: Consumes more CPU
• [ ] Performance regression: Consumes more Memory
• [ ] Breaking backward compatibility

Documentation

• [ ] Affects user behaviors
• [ ] Contains syntax changes
• [ ] Contains variable changes
• [ ] Contains experimental features
• [ ] Changes MySQL compatibility

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

Lightning post-import conflict detection has supported "replace" mode, which is to delete some rows with conflict and reserve other rows that can be kept and not cause conflict anymore.

In comparison, the algorithm for "remove" mode is to delete all the rows that are engaged in conflict.

[codecov[bot]]

Codecov Report

Merging #46763 (cfb7250) into master (cc42e4b) will increase coverage by 0.3830%. Report is 1 commits behind head on master. The diff coverage is 50.5376%.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #46763 +/- ## ================================================ + Coverage 72.2961% 72.6792% +0.3830% ================================================ Files 1352 1373 +21 Lines 400865 407573 +6708 ================================================ + Hits 289810 296221 +6411 - Misses 91856 92557 +701 + Partials 19199 18795 -404 ``` | [Flag](https://app.codecov.io/gh/pingcap/tidb/pull/46763/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | | |---|---|---| | [integration](https://app.codecov.io/gh/pingcap/tidb/pull/46763/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `38.6750% <0.0000%> (?)` | | | [unit](https://app.codecov.io/gh/pingcap/tidb/pull/46763/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `72.2831% <50.5376%> (-0.0131%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap#carryforward-flags-in-the-pull-request-comment) to find out more. | [Components](https://app.codecov.io/gh/pingcap/tidb/pull/46763/components?src=pr&el=components&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | Coverage Δ | | |---|---|---| | [dumpling](https://app.codecov.io/gh/pingcap/tidb/pull/46763/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `53.9913% <ø> (ø)` | | | [parser](https://app.codecov.io/gh/pingcap/tidb/pull/46763/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `84.7651% <ø> (ø)` | | | [br](https://app.codecov.io/gh/pingcap/tidb/pull/46763/components?src=pr&el=component&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=pingcap) | `48.6735% <50.5376%> (-4.4108%)` | :arrow_down: |

[lyzx2001]

/cc @lance6716

[lyzx2001]

/retest

[lyzx2001]

/retest

[lance6716]

/test pull-br-integration-test

[tiprow[bot]]

@lance6716: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1713128658): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1717393335): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/retest

[lance6716]

/test pull-br-integration-test

[tiprow[bot]]

@lance6716: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1718758732): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1720760560): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/retest

[lyzx2001]

/retest

[lyzx2001]

/cc @lance6716

[lyzx2001]

/retest

[lyzx2001]

/retest

[lyzx2001]

/test pull-br-integration-test

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1722789040): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1722951137): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1723069510): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1723632896): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lance6716]

/test pull-br-integration-test

[tiprow[bot]]

@lance6716: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1724750633): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1724802102): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1724874933): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[hawkingrei]

/test all

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1724962749): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1725219469): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: The specified target(s) for /test were not found. The following commands are available to trigger required jobs:

• /test tiprow_fast_test

Use /test all to run all jobs.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1725454355): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/cc @D3Hunter

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1731155854): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1731297360): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[ti-chi-bot[bot]]

[LGTM Timeline notifier]

Timeline:

• 2023-09-25 05:11:08.9700682 +0000 UTC m=+251658.688410417: :ballot_box_with_check: agreed by D3Hunter.
• 2023-09-25 05:11:54.245467872 +0000 UTC m=+251703.963810073: :ballot_box_with_check: agreed by lance6716.

[lyzx2001]

/test pull-br-integration-test

[tiprow[bot]]

@lyzx2001: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1733010985): >/test pull-br-integration-test Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[lyzx2001]

/retest

[tiprow[bot]]

@lyzx2001: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to [this](https://github.com/pingcap/tidb/pull/46763#issuecomment-1733176890): >/retest Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.