search

pingcap / tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.
https://pingcap.com
Apache License 2.0
37.24k stars 5.84k forks source link

lightning Import Large number of tables is too slow #48023

Open aytrack opened 1 year ago

aytrack commented 1 year ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. lightning import about 3.2w tables (2.7T) to tidb cluster

2. What did you expect to see? (Required)

  1. finish import finished with Reasonable time

    3. What did you see instead (Required)

  2. It's takes 73h to import the data
  3. ddl execute during Import data 
    +---------+--------------------+--------------+--------------+--------------+-----------+----------+-----------+---------------------+---------------------+---------------------+--------+
| JOB_ID  | DB_NAME            | TABLE_NAME   | JOB_TYPE     | SCHEMA_STATE | SCHEMA_ID | TABLE_ID | ROW_COUNT | CREATE_TIME         | START_TIME          | END_TIME            | STATE  |
+---------+--------------------+--------------+--------------+--------------+-----------+----------+-----------+---------------------+---------------------+---------------------+--------+
| 1012315 | test               | t4           | create table | public       | 2         | 1012314  | 0         | 2023-10-27 02:32:52 | 2023-10-27 02:32:52 | 2023-10-27 02:32:52 | synced |
| 1008704 | test               | v0           | create view  | public       | 2         | 1008703  | 0         | 2023-10-25 14:19:47 | 2023-10-25 14:19:47 | 2023-10-25 14:19:47 | synced |
| 1008702 | test               | t1           | create table | public       | 2         | 1008701  | 0         | 2023-10-25 14:19:47 | 2023-10-25 14:19:47 | 2023-10-25 14:19:47 | synced |
| 1008700 | test               | t0           | create table | public       | 2         | 1008699  | 0         | 2023-10-25 14:19:46 | 2023-10-25 14:19:46 | 2023-10-25 14:19:46 | synced |
| 1008698 | test               | v0           | drop view    | none         | 2         | 1008693  | 0         | 2023-10-25 14:19:45 | 2023-10-25 14:19:45 | 2023-10-25 14:19:45 | synced |
| 1008697 | test               | t0           | drop table   | none         | 2         | 1008691  | 0         | 2023-10-25 14:19:42 | 2023-10-25 14:19:42 | 2023-10-25 14:19:43 | synced |
| 1008696 | test               | t1           | drop table   | none         | 2         | 1006140  | 0         | 2023-10-25 14:19:32 | 2023-10-25 14:19:32 | 2023-10-25 14:19:33 | synced |
| 1008694 | test               | v0           | create view  | public       | 2         | 1008693  | 0         | 2023-10-25 14:19:17 | 2023-10-25 14:19:17 | 2023-10-25 14:19:17 | synced |
| 1008692 | test               | t0           | create table | public       | 2         | 1008691  | 0         | 2023-10-25 14:19:16 | 2023-10-25 14:19:16 | 2023-10-25 14:19:16 | synced |
| 1006145 | test               | t3           | create table | public       | 2         | 1006144  | 0         | 2023-10-25 09:23:51 | 2023-10-25 09:23:51 | 2023-10-25 09:23:51 | synced |
| 1006143 | test               | t2           | create table | public       | 2         | 1006142  | 0         | 2023-10-25 09:23:50 | 2023-10-25 09:23:50 | 2023-10-25 09:23:50 | synced |
| 1006141 | test               | t1           | create table | public       | 2         | 1006140  | 0         | 2023-10-25 09:23:50 | 2023-10-25 09:23:50 | 2023-10-25 09:23:50 | synced |
| 987070  | test               | ios3         | create table | public       | 2         | 986869   | 0         | 2023-10-23 11:26:50 | 2023-10-23 11:26:50 | 2023-10-23 11:26:51 | synced |
| 986172  | lightning_metadata | task_meta_v2 | create table | public       | 254121    | 986171   | 0         | 2023-10-23 09:22:52 | 2023-10-23 09:22:52 | 2023-10-23 09:22:52 | synced |
| 986170  | co01db99           | co01table66  | create table | public       | 861939    | 986169   | 0         | 2023-10-23 09:22:41 | 2023-10-23 09:22:42 | 2023-10-23 09:22:42 | synced |
| 986168  | co01db99           | co01table32  | create table | public       | 861939    | 986167   | 0         | 2023-10-23 09:22:41 | 2023-10-23 09:22:42 | 2023-10-23 09:22:42 | synced |
| 986166  | co01db99           | co01table69  | create table | public       | 861939    | 986165   | 0         | 2023-10-23 09:22:41 | 2023-10-23 09:22:42 | 2023-10-23 09:22:42 | synced |
| 986164  | co01db99           | co01table58  | create table | public       | 861939    | 986163   | 0         | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | 2023-10-23 09:22:42 | synced |
| 986162  | co01db99           | co01table65  | create table | public       | 861939    | 986161   | 0         | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | synced |
| 986160  | co01db99           | co01table14  | create table | public       | 861939    | 986159   | 0         | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | synced |
| 986158  | co01db99           | co01table24  | create table | public       | 861939    | 986157   | 0         | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | synced |
| 986156  | co01db99           | co01table29  | create table | public       | 861939    | 986155   | 0         | 2023-10-23 09:22:40 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | synced |
| 986154  | co01db99           | co01table33  | create table | public       | 861939    | 986152   | 0         | 2023-10-23 09:22:40 | 2023-10-23 09:22:41 | 2023-10-23 09:22:41 | synced |

    please ignore the ErrChecksumMismatch because there are 171 tables is not empty before import.

    [2023/10/23 08:54:14.153 +00:00] [INFO] [info.go:49] ["Welcome to TiDB-Lightning"] [release-version=v7.5.0-alpha] [git-hash=79db4ec58be3856995d31013e20f3faa57e3a3ac] [git-branch=heads/refs/tags/v7.5.0-alpha] [go-version=go1.21.3] [utc-build-time="2023-10-16 11:39:02"] [race-enabled=false]
[2023/10/26 11:15:06.489 +00:00] [DEBUG] [import.go:1074] ["update checkpoint"] [table=`co01db99`.`co01table66`] [engine_id=2147483647] [new_status=analyzed] []
[2023/10/26 11:15:07.492 +00:00] [ERROR] [import.go:1650] ["restore all tables data failed"] [takeTime=73h1m3.831306634s] [error="[Lighting:Restore:ErrChecksumMismatch]checksum mismatched remote vs local => (checksum: 11885220305671869244 vs 0) (total_kvs: 300000 vs 600000) (total_bytes:14868675 vs 29737350)"]
[2023/10/26 11:15:07.493 +00:00] [INFO] [import.go:1263] ["everything imported, stopping periodic actions"]
[2023/10/26 11:15:07.494 +00:00] [ERROR] [import.go:558] ["run failed"] [step=4] [error="[Lighting:Restore:ErrChecksumMismatch]checksum mismatched remote vs local => (checksum: 11885220305671869244 vs 0) (total_kvs: 300000 vs 600000) (total_bytes:14868675 vs 29737350)"]

4. What is your TiDB version? (Required)

master: 79db4ec58be3856995d31013e20f3faa57e3a3ac

jebter commented 1 year ago

image image

D3Hunter commented 6 months ago

requires config and log to analyze where time is consumed, some related factor