Scale-in CDC to 1 node, owner exited and failed to start

Tammyxia commented 2 years ago

What did you do?

100k table count, cdc 8nodes, 1 changefeed, tikv 6 nodes
Workload is go-ycsb
Scale-out TiKV to 7 nodes
Scale-in TiCDC to 1 node.

What did you expect to see?

No any error

What did you see instead?

CDC owner exited and failed to start from 03/22 00:03, after 12 hours, owner still cannot start. CDC-0 log: [2022/03/21 16:03:29.891 +00:00] [INFO] [capture.go:436] ["run owner exited"] [error="[CDC:ErrCaptureSuicide]capture suicide"] [errorVerbose="[CDC:ErrCaptureSuicide]capture suicide\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/cdc/capture.(Capture).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:471\ngithub.com/pingcap/tiflow/cdc/capture.(Capture).campaignOwner\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:434\ngithub.com/pingcap/tiflow/cdc/capture.(Capture).run.func2\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:298\nruntime.goexit\n\truntime/asm_amd64.s:1571"] error: [2022/03/21 16:03:29.891 +00:00] [INFO] [capture.go:436] ["run owner exited"] [error="[CDC:ErrCaptureSuicide]capture suicide"] [errorVerbose="[CDC:ErrCaptureSuicide]capture suicide\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/cdc/capture.(Capture).runEtcdWorker\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:471\ngithub.com/pingcap/tiflow/cdc/capture.(Capture).campaignOwner\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:434\ngithub.com/pingcap/tiflow/cdc/capture.(Capture).run.func2\n\tgithub.com/pingcap/tiflow/cdc/capture/capture.go:298\nruntime.goexit\n\truntime/asm_amd64.s:1571"]

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)
TiKV
Release Version:   6.0.0-alpha
Edition:           Community
Git Commit Hash:   9a43faf4da20389a4e2c262ee8ab8b369a3bfec4
Git Commit Branch: heads/refs/tags/v6.1.0-nightly
UTC Build Time:    2022-03-19 11:16:46
Rust Version:      rustc 1.60.0-nightly (1e12aef3f 2022-02-13)
Enable Features:   jemalloc mem-profiling portable sse test-engines-rocksdb cloud-aws cloud-gcp cloud-azure
Profile:           dist_release

TiCDC version (execute cdc version):

(paste TiCDC version here)
Release Version: v5.4.0-master
Git Commit Hash: 8fe91733fc2b92de7c9fed6d5981b44b3d9fc3c3
Git Branch: heads/refs/tags/v6.1.0-nightly
UTC Build Time: 2022-03-21 11:13:22
Go Version: go version go1.18 linux/amd64
Failpoint Build: false

Tammyxia commented 2 years ago

CDC process is still there, but [CDC:ErrOwnerNotFound]owner not found"

ps -ef | grep cdc

1 root 3d14 /cdc server --addr=0.0.0.0:8301 --advertise-addr=upstream-ticdc-0.upstream-ticdc-peer.cdc-testbed-tps-662589-1-708.svc:8301 --gc-ttl=86400 --log-file=/var/lib/ticdc/log/ticdc.log --log-level=info --pd=http://upstream-pd:2379 --config=/etc/ticdc/ticdc.toml

Tammyxia commented 2 years ago

goroutines-debug2.tar.gz

overvenus commented 2 years ago

goroutines-debug2.tar.gz

Owner can not be re-elected because owner goroutine is blocked by AsyncClose which is blocked by leveldb sorter close.

goroutine 1068 [select, 214 minutes]:
github.com/pingcap/tiflow/pkg/actor.(*mailbox).SendB(0xc384069ec0, {0x381a080, 0xc000058088}, {0x2, 0x0, {0x0, 0x0, 0x0, {0x0, 0x0}, ...}})
        github.com/pingcap/tiflow/pkg/actor/actor.go:125 +0xdf
github.com/pingcap/tiflow/pkg/actor.(*Router).Broadcast.func1({0xc8e7fb1ad8?, 0x2d838e0?}, {0x2d91e60?, 0xc3a6d34900})
        github.com/pingcap/tiflow/pkg/actor/system.go:292 +0xd9
sync.(*Map).Range(0xc00056cf38?, 0xc8e7fb1b80)
        sync/map.go:347 +0x2aa
github.com/pingcap/tiflow/pkg/actor.(*Router).Broadcast(0xc00056cf30, {0x381a080, 0xc000058088}, {0x2, 0x0, {0x0, 0x0, 0x0, {0x0, 0x0}, ...}})
        github.com/pingcap/tiflow/pkg/actor/system.go:290 +0x12c
github.com/pingcap/tiflow/pkg/actor.(*System).Stop(0xc0003b6380)
        github.com/pingcap/tiflow/pkg/actor/system.go:472 +0x90
github.com/pingcap/tiflow/cdc/sorter/leveldb/system.(*System).Stop(0xc000c3c320)
        github.com/pingcap/tiflow/cdc/sorter/leveldb/system/system.go:200 +0xe5
github.com/pingcap/tiflow/cdc/capture.(*Capture).AsyncClose(0xc001fa4460)
        github.com/pingcap/tiflow/cdc/capture/capture.go:554 +0x19a
github.com/pingcap/tiflow/cdc/capture.(*Capture).run.func2()
        github.com/pingcap/tiflow/cdc/capture/capture.go:300 +0x212
created by github.com/pingcap/tiflow/cdc/capture.(*Capture).run
        github.com/pingcap/tiflow/cdc/capture/capture.go:292 +0x410

https://github.com/pingcap/tiflow/blob/8fe91733fc2b92de7c9fed6d5981b44b3d9fc3c3/cdc/capture/capture.go#L292-L300

Tammyxia commented 2 years ago

Log: https://tcms.pingcap.net/dashboard/executions/plan/662589