pingcap / dumpling

Dumpling is a fast, easy-to-use tool written by Go for dumping data from the database(MySQL, TiDB...) to local/cloud(S3, GCP...) in multifarious formats(SQL, CSV...).
Apache License 2.0
280 stars 85 forks source link

dump: always split TiDB v3.* tables through tidb rowid to save TiDB's memory (#301) #307

Closed ti-chi-bot closed 3 years ago

ti-chi-bot commented 3 years ago

This is an automated cherry-pick of #301

What problem does this PR solve?

fix https://github.com/pingcap/dumpling/issues/295, an extension of https://github.com/pingcap/dumpling/pull/273 In TiDB v3.x:

  1. table INFORMATION_SCHEMA.TIKV_REGION_STATUS doesn't have table_name/table_schema.
  2. doesn't have SHOW TABLE REGIONS statement.
  3. table INFORMATION_SCHEMA.PARTITIONS doesn't actually have data.
  4. doesn't have tidb_decode_key function.
    mysql> show create table information_schema.TIKV_REGION_STATUS;
    +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | Table              | Create Table                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
    +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    | TIKV_REGION_STATUS | CREATE TABLE `TIKV_REGION_STATUS` (
    `REGION_ID` bigint(21) unsigned DEFAULT NULL,
    `START_KEY` text DEFAULT NULL,
    `END_KEY` text DEFAULT NULL,
    `EPOCH_CONF_VER` bigint(21) unsigned DEFAULT NULL,
    `EPOCH_VERSION` bigint(21) unsigned DEFAULT NULL,
    `WRITTEN_BYTES` bigint(21) unsigned DEFAULT NULL,
    `READ_BYTES` bigint(21) unsigned DEFAULT NULL,
    `APPROXIMATE_SIZE` bigint(21) unsigned DEFAULT NULL,
    `APPROXIMATE_KEYS` bigint(21) unsigned DEFAULT NULL
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin |
    +--------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
    1 row in set (0.00 sec)

What is changed and how it works?

Split chunks by PKIsHandle for TiDB v3.*. Make sure TiDB won't OOM through splitting tables by rowids. We do some staff by our own to solve the problem that tidb lacks some info.

Check List

Tests

Related changes

Release note

ti-chi-bot commented 3 years ago

[REVIEW NOTIFICATION]

This pull request has been approved by:

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment. After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review. Reviewer can cancel approval by submitting a request changes review.
lichunzhu commented 3 years ago

/merge

ti-chi-bot commented 3 years ago

This pull request has been accepted and is ready to merge.

Commit hash: 3712e6f3c8fe1a7ec75261b98c0bfcfdc5fe18b6