Refine PITR, make it production-ready

Introduce

PITR is an ecosystem tool for TiDB Binlog. By preprocessing the incremental backup file of TiDB, PITR merged the changes of the same line of data to produce a new, lighter incremental backup file, which greatly reduced the Time of incremental backup Recovery and realized fast-pitr (Fast Point in Time Recovery).

For example

There is a table t1, it's schema is: create table t1 (id int primary key, name varchar(24)). And now we execute four SQLs in TiDB:

insert into t1 values(1, “a”);
insert into t1 values(2, “b”);
update t1 set name = “c” where id = 1;
delete from t1 where id = 2;

These SQLs will generate four binlog, restore binlog using Reparo tool data to downstream will execute four SQLs in downstream database. These binlogs are actually can merged to generate an insert into t1 values(1, "c"); This will save a quarter as much space as before and restore the files four times as fast. We can think of it simply: the binlog file produced by Drainer is compressed/preprocessed by PITR.

Current Situation

PITR is a Hackathon project, so it only implements the basic functionality, has some known problems, and lacks testing, so there may be more unknown problems.

Bug

PITR needs to maintain the table structure information (to parse the binlog data) but will report an error if there is no database information in the DDL.

Performance

PITR retrieves TiDB's historical DDL information to build the table structure of the initial state, and executes the DDL in PITR's built-in TiDB to retrieve the table structure. If the history of DDL is large, the initialization state can be very slow. (defective design)

Test

Unit test coverage was 63.8%
No integration testing

Usability

PITR does not output meta information, such as the location of the file to run the processing, or the time period during which binlog was processed (the binlog commit ts at the beginning and the binlog commit ts at the end).

TODO List

[x] Fix the bug that will report an error if there is no database information in the DDL.
[x] Solve the performance problem for DDLhandle.
[x] Supplementary unit tests, increased to above 75%.
[x] Add integration testing to cover basic usage scenarios.
[x] Refine PITR's usability, save PITR's meta information.
[x] Fix problems found in tests.

Difficulty

Normal

Score

3(developer) 7(day) 300(score)

Quick Start

Deploy TiDB and Binlog cluster according to Deploy TiDB Binlog, note that the db-type of Drainer should be file.
Execute some SQLs in TiDB to generate binlog. You can check the binlog file generated by Drainer.
Use PITR to merge binlog file.

lvleiice / Better-PITR