search

pingcap / tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.
https://pingcap.com
Apache License 2.0
37.28k stars 5.84k forks source link

Import data using Lightning from cloud storage failed due to "scan regions from start-key" error #50333

Open hongshaoyang opened 10 months ago

hongshaoyang commented 10 months ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

Import data using Lightning from cloud storage using Physical Import Mode (backend = local)

2. What did you expect to see? (Required)

Lightning succeeds.

3. What did you see instead (Required)

Lightning failed due to error from TiDB server.

tidb lightning encountered error: [Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region
[2024/01/11 08:42:09.464 +00:00] [ERROR] [main.go:103] [""tidb lightning encountered error stack info""] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""] [errorVerbose=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/normalize.go:164\ngithub.com/pingcap/tidb/br/pkg/lightning/common.NormalizeOrWrapErr\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/common/errors.go:222\ngithub.com/pingcap/tidb/br/pkg/lightning/importer.(*Controller).importTables.func6\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/import.go:1656\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598""]
[2024/01/11 08:42:09.438 +00:00] [ERROR] [backend.go:350] [""import failed""] [engineTag=<import-and-reset>] [engineUUID=317d5a82-b901-5f31-a8fb-9b4223031f4f] [retryCnt=0] [takeTime=1m26.668744789s] [error=""scan regions from start-key:7480000000000023FF645F698000000000FF000001010E290E29FF0E320E4AFF0E6D0EFF6D0E2C0E2EFF0221FF0E2C0E2C0E29FF0EFF3102210E2D0E29FFFF0E310E2F02210E32FFFF0E6D0E310E3202FF21FF0E330E4A0E2CFF0E6DFF0E4A0E330EFF320E60FF0E6D0E4AFF0E8B0E29FF000000FF0000000000F70000FD, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:09.438 +00:00] [ERROR] [import.go:2079] [""importing large engines for disk quota failed""] [diskSize=301382974674] [memSize=0] [quota=268435456000] [largeEnginesCount=1] [inProgressLargeEnginesCount=0] [takeTime=1m26.668819276s] [error=""scan regions from start-key:7480000000000023FF645F698000000000FF000001010E290E29FF0E320E4AFF0E6D0EFF6D0E2C0E2EFF0221FF0E2C0E2C0E29FF0EFF3102210E2D0E29FFFF0E310E2F02210E32FFFF0E6D0E310E3202FF21FF0E330E4A0E2CFF0E6DFF0E4A0E330EFF320E60FF0E6D0E4AFF0E8B0E29FF000000FF0000000000F70000FD, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.686 +00:00] [ERROR] [import.go:521] [""run failed""] [step=4] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.686 +00:00] [ERROR] [import.go:531] [""the whole procedure failed""] [takeTime=5h17m20.651341731s] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.686 +00:00] [ERROR] [import.go:168] [""tables failed to be imported""] [count=1]
[2024/01/11 08:42:02.686 +00:00] [ERROR] [import.go:170] [-] [table=`assets_proxy_prod_gl`.`assets`] [status=imported] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.680 +00:00] [ERROR] [import.go:1619] [""restore all tables data failed""] [takeTime=5h17m19.933853996s] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.438 +00:00] [ERROR] [import.go:1657] [""restore table failed""] [table=`assets_proxy_prod_gl`.`assets`] [takeTime=5h17m19.668266409s] [error=""[Lightning:Restore:ErrRestoreTable]restore table `assets_proxy_prod_gl`.`assets` failed: scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.438 +00:00] [ERROR] [table_import.go:1205] [""import and cleanup engine failed""] [engineTag=`assets_proxy_prod_gl`.`assets`:0] [engineUUID=ca27de63-4c5f-56a2-9852-a309b567dfbe] [takeTime=2m14.243945508s] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.438 +00:00] [ERROR] [table_import.go:477] [""import whole table failed""] [table=`assets_proxy_prod_gl`.`assets`] [takeTime=5h16m54.536504671s] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]
[2024/01/11 08:42:02.438 +00:00] [ERROR] [import.go:1653] [""failed to import table""] [table=`assets_proxy_prod_gl`.`assets`] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""] [errorVerbose=""[BR:PD:ErrPDBatchScanRegion]batch scan region\nscan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled\ngithub.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:108\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:57\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetryV2[...]\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:75\ngithub.com/pingcap/tidb/br/pkg/utils.WithRetry\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/utils/retry.go:56\ngithub.com/pingcap/tidb/br/pkg/restore/split.PaginateScanRegion\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/restore/split/split.go:101\ngithub.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).generateJobForRange\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1218\ngithub.com/pingcap/tidb/br/pkg/lightning/backend/local.(*Backend).generateAndSendJob.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/local/local.go:1150\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/go/pkg/mod/golang.org/x/sync@v0.2.0/errgroup/errgroup.go:75\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598""]
[2024/01/11 08:42:02.428 +00:00] [ERROR] [backend.go:350] [""import failed""] [engineTag=`assets_proxy_prod_gl`.`assets`:0] [engineUUID=ca27de63-4c5f-56a2-9852-a309b567dfbe] [retryCnt=0] [takeTime=2m14.234574581s] [error=""scan regions from start-key:7480000000000023FF645F728000000201FFA91E640000000000FA, err: rpc error: code = Canceled desc = context canceled: [BR:PD:ErrPDBatchScanRegion]batch scan region""]

4. What is your TiDB version? (Required)

v7.1.2

hongshaoyang commented 10 months ago

Attached log of lightning here

lance6716 commented 10 months ago

generally "context canceled" does not mean the root cause. Can you find other errors in the log?

hongshaoyang commented 10 months ago

I'm unable to find other errors in the log.