search

pingcap / tidb

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://tidbcloud.com/free-trial
https://pingcap.com
Apache License 2.0
36.27k stars 5.72k forks source link

fast reorg checkpoint has wrong prerequisite #53009

Closed lance6716 closed 1 week ago

lance6716 commented 2 weeks ago

If the checkpoint record shows the it is generated by current node

https://github.com/pingcap/tidb/blob/c24dca58f5ee68941a656ac898136eaf63070703/pkg/ddl/ingest/checkpoint.go#L307-L308

we skip the data whose key is smaller that "local storage key watermark" and assume the data is already stay in local storage.

https://github.com/pingcap/tidb/blob/c24dca58f5ee68941a656ac898136eaf63070703/pkg/ddl/ingest/checkpoint.go#L128-L137

However this is not reliable, especially we are using /tmp folder to store the data https://docs.pingcap.com/tidb/stable/command-line-flags-for-tidb-configuration#--temp-dir. If the node restarts in many OS /tmp folder will be cleaned.