pingcap / tidb

TiDB is an open-source, cloud-native, distributed, MySQL-Compatible database for elastic scale and real-time analytics. Try AI-powered Chat2Query free at : https://www.pingcap.com/tidb-serverless/
https://pingcap.com
Apache License 2.0
36.88k stars 5.81k forks source link

Implement the toolkit for managing `Migration`s. #55661

Open YuJuncen opened 2 weeks ago

YuJuncen commented 2 weeks ago

Enhancement

We have designed the migration system for compacting log backup. Generally, every modifications over the "stabilized" storage must be done by appending a "migration".

     +-Modifications over those files                 
     | Must be done by Appending a migration.         
     |                                                
+----|--------------+-------------+                   
| stabilized        | Appending   |                   
|-------------------|-------------+                   
|                    \                                
 \                   |                                
 |Truncated TS        \Resolved TS                    

Migrations are idempotent batch modifications (adding a compaction, delete a file, etc..) to the backup files. You may check the protocol buffer message Migration for more details. Idempontence is important for migrations, as they may be executed multi times due to retry or racing.

YuJuncen commented 2 weeks ago

We need to implement the toolkit for the Migration storage.