RFC + Support of BACKUP and RESTORE statements

kennytm commented 4 years ago

What problem does this PR solve?

Support running BR inside TiDB directly.

What is changed and how it works?

Recognize the new *ast.BRIEStmt in pingcap/parser#746, and forward to the library functions in BR. When we execute

BACKUP DATABASE `tpcc` TO 'local:///tmp/storage/';

TiDB will spawn a new BR manager which backs up the database tpcc into the provided storage. The query blocks until backup completes. Returns an empty set on success:

MySQL [tpcc]> backup database tpcc to 'local:///tmp/br_tpcc_32';
Empty set (58.453 sec)

and returns an error on failure:

MySQL [tpcc]> backup table tpcc.stock to 'local:///tmp/br_tpcc_30';
ERROR 8124 (HY000): Backup failed: backup meta exists, may be some backup files in the path already

BRIE tasks must be executed sequentially. Currently, for simplicity, tasks are queued in the local server only. In the future we make the entire cluster share the same queue.

Use SHOW BACKUP / SHOW RESTORE in another session to list the tasks

MySQL [(none)]> show backup;
+-------------------------+---------+-------------------+---------------------+---------------------+------+
| Storage                 | State   | Progress          | Init_time           | Step_start_time     | ID   |
+-------------------------+---------+-------------------+---------------------+---------------------+------+
| local:///tmp/br_tpcc_30 | Backup  | 98.38709677419355 | 2020-04-12 23:09:03 | 2020-04-12 23:09:25 |    3 |
| local:///tmp/br_tpcc_30 | Wait    |                 0 | 2020-04-12 23:09:48 | 2020-04-12 23:09:48 |    4 |
+-------------------------+---------+-------------------+---------------------+---------------------+------+

Use KILL TIDB QUERY n to cancel a task.

Note: Currently running RESTORE may make the tables enter a "non-ACID" state where the backup archives are partially ingested. Maybe we need to pessimistically lock the entire database?

Note: No test cases yet. What to do?

Check List

Tests

Manual test (add detailed scripts or steps below)
- Running backup on ~~a simple table~~ W=30 TPC-C database (2 GB), drop it, and run restore from the archive.

Code changes

Side effects

Related changes

Need to update the documentation

Release note

Added the BACKUP statement to create a logical backup archive.
~~Added the RESTORE statement to restore from the backup archive.~~ (don't include into release note yet, do so after the entire feature is complete.)

kennytm commented 4 years ago

/merge

sre-bot commented 4 years ago

/run-all-tests

sre-bot commented 4 years ago

@kennytm merge failed.

kennytm commented 4 years ago

https://internal.pingcap.net/idc-jenkins/blue/organizations/jenkins/tidb_ghpr_integration_copr_test/detail/tidb_ghpr_integration_copr_test/4446/pipeline

[2020-04-30T11:28:53.945Z] Statement: #813 -  SELECT SHA1( `col_smallint_key` ) AS field1, ENCODE( `col_bit_key`, `col_bit` ) AS field2 FROM `table1_int_autoinc` WHERE ! `col_float_key` ORDER BY field1, field2 LIMIT 4 /* QNO 815 CON_ID 152 */ ;

[2020-04-30T11:28:53.945Z] NoPushDown Output: 

[2020-04-30T11:28:53.945Z] field1   field2

[2020-04-30T11:28:53.945Z] 

[2020-04-30T11:28:53.945Z] 

[2020-04-30T11:28:53.945Z] WithPushDown Output: 

[2020-04-30T11:28:53.945Z] field1   field2

[2020-04-30T11:28:53.945Z] fcaf7c134f49c65f2c5765d81c181f3334cefe18 

[2020-04-30T11:28:53.945Z] 

[2020-04-30T11:28:53.945Z] 

[2020-04-30T11:28:53.945Z] 

[2020-04-30T11:28:53.945Z] 

[2020-04-30T11:28:53.945Z] NoPushDown Plan: 

[2020-04-30T11:28:53.945Z] id   estRows task    access object   operator info

[2020-04-30T11:28:53.945Z] Projection_7 0.00    root        sha1(cast(push_down_test_db.table1_int_autoinc.col_smallint_key, var_string(20)))->Column#62, encode(cast(push_down_test_db.table1_int_autoinc.col_bit_key, var_string(20)), cast(push_down_test_db.table1_int_autoinc.col_bit, var_string(20)))->Column#63

[2020-04-30T11:28:53.945Z] └─Projection_23  0.00    root        push_down_test_db.table1_int_autoinc.col_smallint_key, push_down_test_db.table1_int_autoinc.col_bit, push_down_test_db.table1_int_autoinc.col_bit_key, push_down_test_db.table1_int_autoinc.col_float_key

[2020-04-30T11:28:53.945Z]   └─TopN_8   0.00    root        Column#66:asc, Column#67:asc, offset:0, count:4

[2020-04-30T11:28:53.945Z]     └─Projection_24  0.00    root        push_down_test_db.table1_int_autoinc.col_smallint_key, push_down_test_db.table1_int_autoinc.col_bit, push_down_test_db.table1_int_autoinc.col_bit_key, push_down_test_db.table1_int_autoinc.col_float_key, sha1(cast(push_down_test_db.table1_int_autoinc.col_smallint_key, var_string(20)))->Column#66, encode(cast(push_down_test_db.table1_int_autoinc.col_bit_key, var_string(20)), cast(push_down_test_db.table1_int_autoinc.col_bit, var_string(20)))->Column#67

[2020-04-30T11:28:53.945Z]       └─TableReader_13   0.00    root        data:Selection_12

[2020-04-30T11:28:53.945Z]         └─Selection_12   0.00    cop[tikv]       not(istrue(push_down_test_db.table1_int_autoinc.col_float_key))

[2020-04-30T11:28:53.945Z]           └─TableFullScan_11 1.00    cop[tikv]   table:table1_int_autoinc    keep order:false, stats:pseudo

[2020-04-30T11:28:53.945Z] 

[2020-04-30T11:28:53.945Z] 

[2020-04-30T11:28:53.945Z] WithPushDown Plan: 

[2020-04-30T11:28:53.945Z] id   estRows task    access object   operator info

[2020-04-30T11:28:53.945Z] Projection_7 4.00    root        sha1(cast(push_down_test_db.table1_int_autoinc.col_smallint_key, var_string(20)))->Column#62, encode(cast(push_down_test_db.table1_int_autoinc.col_bit_key, var_string(20)), cast(push_down_test_db.table1_int_autoinc.col_bit, var_string(20)))->Column#63

[2020-04-30T11:28:53.945Z] └─Projection_23  4.00    root        push_down_test_db.table1_int_autoinc.col_smallint_key, push_down_test_db.table1_int_autoinc.col_bit, push_down_test_db.table1_int_autoinc.col_bit_key, push_down_test_db.table1_int_autoinc.col_float_key

[2020-04-30T11:28:53.945Z]   └─TopN_9   4.00    root        Column#66:asc, Column#67:asc, offset:0, count:4

[2020-04-30T11:28:53.945Z]     └─Projection_24  20.00   root        push_down_test_db.table1_int_autoinc.col_smallint_key, push_down_test_db.table1_int_autoinc.col_bit, push_down_test_db.table1_int_autoinc.col_bit_key, push_down_test_db.table1_int_autoinc.col_float_key, sha1(cast(push_down_test_db.table1_int_autoinc.col_smallint_key, var_string(20)))->Column#66, encode(cast(push_down_test_db.table1_int_autoinc.col_bit_key, var_string(20)), cast(push_down_test_db.table1_int_autoinc.col_bit, var_string(20)))->Column#67

[2020-04-30T11:28:53.945Z]       └─IndexLookUp_16   20.00   root        

[2020-04-30T11:28:53.945Z]         ├─IndexRangeScan_14(Build)   20.00   cop[tikv]   table:table1_int_autoinc, index:col_float_key(col_float_key)    range:[NULL,NULL], [0,0], keep order:false, stats:pseudo

[2020-04-30T11:28:53.945Z]         └─TableRowIDScan_15(Probe)   20.00   cop[tikv]   table:table1_int_autoinc    keep order:false, stats:pseudo

kennytm commented 4 years ago

/merge

sre-bot commented 4 years ago

/run-all-tests

kennytm commented 4 years ago

/run-cherry-picker

sre-bot commented 4 years ago

cherry pick to release-4.0 in PR #16960

pingcap / tidb