PITR is an ecosystem tool for TiDB Binlog. By preprocessing the incremental backup file of TiDB, PITR merged the changes of the same line of data to produce a new, lighter incremental backup file, which greatly reduced the Time of incremental backup Recovery and realized fast-pitr (Fast Point in Time Recovery).
For example
There is a table t1, it's schema is: create table t1 (id int primary key, name varchar(24)). And now we execute four SQLs in TiDB:
insert into t1 values(1, “a”);
insert into t1 values(2, “b”);
update t1 set name = “c” where id = 1;
delete from t1 where id = 2;
These SQLs will generate four binlog, restore binlog using Reparo tool data to downstream will execute four SQLs in downstream database. These binlogs are actually can merged to generate an insert into t1 values(1, "c"); This will save a quarter as much space as before and restore the files four times as fast. We can think of it simply: the binlog file produced by Drainer is compressed/preprocessed by PITR.
Current Situation
PITR is a Hackathon project, so it only implements the basic functionality, has some known problems, and lacks testing, so there may be more unknown problems. We need to solve the below problems, and make PITR production-ready.
Bug
PITR needs to maintain the table structure information (to parse the binlog data) but will report an error if there is no database information in the DDL.
Performance
PITR retrieves TiDB's historical DDL information to build the table structure of the initial state, and executes the DDL in PITR's built-in TiDB to retrieve the table structure. If the history of DDL is large, the initialization state can be very slow. (defective design)
Test
Unit test coverage was 63.8%
No integration testing
Usability
PITR does not output meta information, such as the location of the file to run the processing, or the time period during which binlog was processed (the binlog commit ts at the beginning and the binlog commit ts at the end).
Introduce
github: https://github.com/lvleiice/Better-PITR
PITR is an ecosystem tool for TiDB Binlog. By preprocessing the incremental backup file of TiDB, PITR merged the changes of the same line of data to produce a new, lighter incremental backup file, which greatly reduced the Time of incremental backup Recovery and realized fast-pitr (Fast Point in Time Recovery).
For example
There is a table
t1
, it's schema is:create table t1 (id int primary key, name varchar(24))
. And now we execute four SQLs in TiDB:These SQLs will generate four binlog, restore binlog using Reparo tool data to downstream will execute four SQLs in downstream database. These binlogs are actually can merged to generate an
insert into t1 values(1, "c")
; This will save a quarter as much space as before and restore the files four times as fast. We can think of it simply: the binlog file produced by Drainer is compressed/preprocessed by PITR.Current Situation
PITR is a Hackathon project, so it only implements the basic functionality, has some known problems, and lacks testing, so there may be more unknown problems. We need to solve the below problems, and make PITR production-ready.
Bug
Performance
Test
Usability
Estimated Time
3(Developers) * 7(days)