search

pingcap / tidb

TiDB - the open-source, cloud-native, distributed SQL database designed for modern applications.
https://pingcap.com
Apache License 2.0
37.28k stars 5.84k forks source link

TiDB panic during lightning logical import #47515

Open fubinzh opened 1 year ago

fubinzh commented 1 year ago

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

  1. lightning tidb backend import

2. What did you expect to see? (Required)

  1. Import should succeed

3. What did you see instead (Required)

Lightning import failed, tidb panic seen

[2023/10/10 04:46:39.934 +00:00] [ERROR] [table_import.go:526] ["import whole table failed"] [table=`bigc`.`t`] [takeTime=40m52.405135307s] [error="[`bigc`.`t`] batch write rows reach max retry 3 and still failed: dial tcp 10.233.53.181:4000: connect: connection refused"]
[2023/10/10 04:46:39.934 +00:00] [ERROR] [import.go:1684] ["failed to import table"] [table=`bigc`.`t`] [error="[`bigc`.`t`] batch write rows reach max retry 3 and still failed: dial tcp 10.233.53.181:4000: connect: connection refused"] [errorVerbose="dial tcp 10.233.53.181:4000: connect: connection refused\ngithub.com/pingcap/errors.AddStack\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/errors.go:174\ngithub.com/pingcap/errors.Trace\n\t/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20221009092201-b66cddb77c32/juju_adaptor.go:15\ngithub.com/pingcap/tidb/br/pkg/lightning/backend/tidb.(*tidbBackend).execStmts\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/tidb/tidb.go:761\ngithub.com/pingcap/tidb/br/pkg/lightning/backend/tidb.(*tidbBackend).WriteBatchRowsToDB\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/tidb/tidb.go:667\ngithub.com/pingcap/tidb/br/pkg/lightning/backend/tidb.(*tidbBackend).WriteRows\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/tidb/tidb.go:617\ngithub.com/pingcap/tidb/br/pkg/lightning/backend/tidb.(*Writer).AppendRows\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/backend/tidb/tidb.go:862\ngithub.com/pingcap/tidb/br/pkg/lightning/importer.(*chunkProcessor).deliverLoop.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/chunk_process.go:657\ngithub.com/pingcap/tidb/br/pkg/lightning/importer.(*chunkProcessor).deliverLoop\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/chunk_process.go:681\ngithub.com/pingcap/tidb/br/pkg/lightning/importer.(*chunkProcessor).process.func1\n\t/home/jenkins/agent/workspace/build-common/go/src/github.com/pingcap/br/br/pkg/lightning/importer/chunk_process.go:198\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\n[`bigc`.`t`] batch write rows reach max retry 3 and still failed"]
[2023/10/10 04:46:39.935 +00:00] [ERROR] [import.go:1688] ["restore table failed"] [table=`bigc`.`t`] [takeTime=40m52.40835841s] [error="[Lightning:Restore:ErrRestoreTable]restore table `bigc`.`t` failed: [`bigc`.`t`] batch write rows reach max retry 3 and still failed: dial tcp 10.233.53.181:4000: connect: connection refused"]

kubectl --kubeconfig kubeconfig.yml -n cdc-kafka-scale-cdc-scale-tps-3030040-1-444 logs -p upstream-tidb-1
start tidb-server ...
/tidb-server --store=tikv --advertise-address=upstream-tidb-1.upstream-tidb-peer.cdc-kafka-scale-cdc-scale-tps-3030040-1-444.svc --host=[0.0.0.0](http://0.0.0.0/) --path=upstream-pd:2379 --config=/etc/tidb/tidb.toml

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x128 pc=0x4c3a1e9]

goroutine 23368311 [running]:
[github.com/pingcap/tidb/server.(*rpcServer).KvPrewrite](http://github.com/pingcap/tidb/server.(*rpcServer).KvPrewrite)(0x5601b40?, {0x60b7ac0?, 0xc0163df4d0?}, 0x0?)
        <autogenerated>:1 +0x29
[github.com/pingcap/kvproto/pkg/tikvpb._Tikv_KvPrewrite_Handler](http://github.com/pingcap/kvproto/pkg/tikvpb._Tikv_KvPrewrite_Handler)({0x56f8880?, 0xc003e61740}, {0x60b7ac0, 0xc0163df4d0}, 0xc00296b880, 0x0)
        /go/pkg/mod/[github.com/pingcap/kvproto@v0.0.0-20230904082117-ecdbf1f8c130/pkg/tikvpb/tikvpb.pb.go:2693](http://github.com/pingcap/kvproto@v0.0.0-20230904082117-ecdbf1f8c130/pkg/tikvpb/tikvpb.pb.go:2693) +0x16c
[google.golang.org/grpc.(*Server).processUnaryRPC](http://google.golang.org/grpc.(*Server).processUnaryRPC)(0xc001dd2b40, {0x60d63a0, 0xc009ec51e0}, 0xc00b56fe60, 0xc003e61860, 0x873b9f0, 0x0)
        /go/pkg/mod/[google.golang.org/grpc@v1.54.0/server.go:1345](http://google.golang.org/grpc@v1.54.0/server.go:1345) +0xde7
[google.golang.org/grpc.(*Server).handleStream](http://google.golang.org/grpc.(*Server).handleStream)(0xc001dd2b40, {0x60d63a0, 0xc009ec51e0}, 0xc00b56fe60, 0x0)
        /go/pkg/mod/[google.golang.org/grpc@v1.54.0/server.go:1722](http://google.golang.org/grpc@v1.54.0/server.go:1722) +0x9e7
[google.golang.org/grpc.(*Server).serveStreams.func1.2()](http://google.golang.org/grpc.(*Server).serveStreams.func1.2())
        /go/pkg/mod/[google.golang.org/grpc@v1.54.0/server.go:966](http://google.golang.org/grpc@v1.54.0/server.go:966) +0x8d
created by [google.golang.org/grpc.(*Server).serveStreams.func1](http://google.golang.org/grpc.(*Server).serveStreams.func1) in goroutine 23368202
        /go/pkg/mod/[google.golang.org/grpc@v1.54.0/server.go:964](http://google.golang.org/grpc@v1.54.0/server.go:964) +0x246

4. What is your TiDB version? (Required)

[root@upstream-tidb-0 /]# /tidb-server -V Release Version: v7.4.0 Edition: Community Git Commit Hash: 04ac2000e640cee9685ea288c5a2851d765ed28e Git Branch: heads/refs/tags/v7.4.0 UTC Build Time: 2023-10-09 09:17:37 GoVersion: go1.21.1 Race Enabled: false Check Table Before Drop: false Store: unistore

fubinzh commented 1 year ago

/label affects-7.4

fubinzh commented 1 year ago

/component tidb

ti-chi-bot[bot] commented 1 year ago

@fubinzh: The label(s) component/tidb cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/pingcap/tidb/issues/47515#issuecomment-1754668747): >/component tidb Instructions for interacting with me using PR comments are available [here](https://prow.tidb.net/command-help). If you have questions or suggestions related to my behavior, please file an issue against the [ti-community-infra/tichi](https://github.com/ti-community-infra/tichi/issues/new?title=Prow%20issue:) repository.
fubinzh commented 6 months ago

This issue is not seen during regression for last 5 months.