search

pingcap / tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.
https://pingcap.com
Apache License 2.0
37.28k stars 5.84k forks source link

Reuse "startTS" for lightning's duplicate detection #51312

Open lance6716 opened 8 months ago

lance6716 commented 8 months ago

Enhancement

ref https://github.com/tikv/tikv/issues/16533

In above request, lightning must use GetTS to retrieve commitTS after nofity RPC returns in ingestion step. TiKV may restart and lightning should retry from notify RPC in that case, which causes an encoded KV be ingested with multiple different commitTS. (Please note that we reuse the commitTS concept to let PiTR have unified logic to process ingestion and transaction, although in ingestion the commitTS is not related to 2PC)

In other words, only startTS can be used when running duplicate detection. (Again this is only to reuse the existing concepts, not a real 2PC)

To have a fixed startTS for an encoded KV, it should be decided before ingestion step. Currently for lightning local engine, there's a TS saved in engine's DB when open/reset engine, and this TS works well as the commitTS in ingestion step. Similarly for external engine, we need to save a TS in external engine's metadata, this is https://github.com/pingcap/tidb/issues/46986

Considering lightning/DDL crash case. lightning/DDL has asynchronized checkpoint. For local engine, when disk quota is reached, lightning/DDL will ingest the data to TiKV and reset the local engine. During reset the local engine its startTS is also changed. If lightning/DDL crashes before saving the checkpoint, after resuming from checkpoint, same KV will be encoded and ingested with a different startTS. To avoid problem in this case, TiKV's duplicate detection should tolerate KV of the same key and the same value.

For external engine, write&encode step does not pipeline with ingest step. There's no such problem.

D3Hunter commented 8 months ago

If lightning/DDL crashes before saving the checkpoint, after resuming from checkpoint, same KV will be encoded and ingested with a different startTS.

if ts is reset and saved into local pebble db, we will get it there even on restart

lance6716 commented 8 months ago

If lightning/DDL crashes before saving the checkpoint, after resuming from checkpoint, same KV will be encoded and ingested with a different startTS.

if ts is reset and saved into local pebble db, we will get it there even on restart

The TS is reset, but it's a new TS. Because checkpoint is not updated before crash, lightning/DDL will read the old data from outdated checkpoint again, encode and write into local engine, and import with new TS. TiKV will see a same key value pair in two different TS.