search

pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
Apache License 2.0
430 stars 286 forks source link

capture info cannot be removed in time in the upgrade scenario. #6638

Open 3AceShowHand opened 2 years ago

3AceShowHand commented 2 years ago 
[2022/08/05 15:03:26.102 +08:00] [INFO] [version.go:47] ["Welcome to Change Data Capture (CDC)"] [release-version=v6.3.0-master] [git-hash=5bd84aea209e9f962e080d2fb2a005a79773591e] [git-branch=master] [utc-build-time="2022-08-04 11:50:00"] [go-version="go version go1.18 linux/amd64"] [failpoint-build=false]
[2022/08/05 15:03:27.138 +08:00] [INFO] [capture.go:263] ["init capture"] [captureID=34cfeb5f-1112-4016-a14a-54c112308120] [captureAddr=lingjin-4.jinling.tispace:8300]
[2022/08/05 15:03:27.143 +08:00] [INFO] [reactor_state.go:88] ["remote capture online"] [captureID=34cfeb5f-1112-4016-a14a-54c112308120] [info="{\"id\":\"34cfeb5f-1112-4016-a14a-54c112308120\",\"address\":\"lingjin-4.jinling.tispace:8300\",\"version\":\"v6.3.0-master\"}"]
[2022/08/05 15:03:27.143 +08:00] [INFO] [reactor_state.go:88] ["remote capture online"] [captureID=c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9] [info="{\"id\":\"c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9\",\"address\":\"lingjin-4.jinling.tispace:8300\",\"version\":\"v6.2.0\"}"]
[2022/08/05 15:03:27.143 +08:00] [INFO] [reactor_state.go:88] ["remote capture online"] [captureID=dbde680f-3122-4b2b-bbc0-66dd7d3fe68c] [info="{\"id\":\"dbde680f-3122-4b2b-bbc0-66dd7d3fe68c\",\"address\":\"lingjin-3.jinling.tispace:8300\",\"version\":\"v6.2.0\"}"]
[2022/08/05 15:03:35.484 +08:00] [INFO] [reactor_state.go:72] ["remote capture offline"] [captureID=c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9] [info="{\"id\":\"c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9\",\"address\":\"lingjin-4.jinling.tispace:8300\",\"version\":\"v6.2.0\"}"]
3AceShowHand commented 2 years ago

the owner instance log:

[2022/08/05 15:03:26.926 +08:00] [WARN] [client.go:168] ["peer message client detected error, restarting"] [error="rpc error: code = Unavailable desc = CDC capture is not running"] [errorVerbose="rpc error: code = Unavailable desc = CDC capture is not running\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.Trace\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/juju_adaptor.go:15\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).runRx\n\tgithub.com/pingcap/tiflow/pkg/p2p/client.go:349\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).run.func2\n\tgithub.com/pingcap/tiflow/pkg/p2p/client.go:205\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/08/05 15:03:27.140 +08:00] [INFO] [reactor_state.go:88] ["remote capture online"] [captureID=34cfeb5f-1112-4016-a14a-54c112308120] [info="{\"id\":\"34cfeb5f-1112-4016-a14a-54c112308120\",\"address\":\"lingjin-4.jinling.tispace:8300\",\"version\":\"v6.3.0-master\"}"]
[2022/08/05 15:03:27.140 +08:00] [WARN] [owner.go:409] ["the capture version is different with the owner"] [capture="{\"id\":\"34cfeb5f-1112-4016-a14a-54c112308120\",\"address\":\"lingjin-4.jinling.tispace:8300\",\"version\":\"v6.3.0-master\"}"] [ownerVer=v6.2.0]
[2022/08/05 15:03:27.142 +08:00] [INFO] [reactor_state.go:88] ["remote capture online"] [captureID=34cfeb5f-1112-4016-a14a-54c112308120] [info="{\"id\":\"34cfeb5f-1112-4016-a14a-54c112308120\",\"address\":\"lingjin-4.jinling.tispace:8300\",\"version\":\"v6.3.0-master\"}"]
[2022/08/05 15:03:27.412 +08:00] [INFO] [helper.go:78] ["got signal, prepare to shutdown"] [signal=terminated]
[2022/08/05 15:03:27.926 +08:00] [INFO] [client.go:117] ["peer message client exited"] [addr=lingjin-4.jinling.tispace:8300] [captureID=c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9] [error="[CDC:ErrPeerMessageClientPermanentFail]peer-to-peer message client has failed permanently, no need to reconnect: [CDC:ErrPeerMessageReceiverMismatch]peer-to-peer message receiver is a mismatch: expected 34cfeb5f-1112-4016-a14a-54c112308120, got c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9"] [errorVerbose="[CDC:ErrPeerMessageClientPermanentFail]peer-to-peer message client has failed permanently, no need to reconnect: [CDC:ErrPeerMessageReceiverMismatch]peer-to-peer message receiver is a mismatch: expected 34cfeb5f-1112-4016-a14a-54c112308120, got c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).runRx\n\tgithub.com/pingcap/tiflow/pkg/p2p/client.go:355\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).run.func2\n\tgithub.com/pingcap/tiflow/pkg/p2p/client.go:205\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/08/05 15:03:27.926 +08:00] [WARN] [message_router.go:145] ["p2p client exited with error"] [addr=lingjin-4.jinling.tispace:8300] [targetCapture=c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9] [error="[CDC:ErrPeerMessageClientPermanentFail]peer-to-peer message client has failed permanently, no need to reconnect: [CDC:ErrPeerMessageReceiverMismatch]peer-to-peer message receiver is a mismatch: expected 34cfeb5f-1112-4016-a14a-54c112308120, got c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9"] [errorVerbose="[CDC:ErrPeerMessageClientPermanentFail]peer-to-peer message client has failed permanently, no need to reconnect: [CDC:ErrPeerMessageReceiverMismatch]peer-to-peer message receiver is a mismatch: expected 34cfeb5f-1112-4016-a14a-54c112308120, got c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9\ngithub.com/pingcap/errors.AddStack\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\tgithub.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).runRx\n\tgithub.com/pingcap/tiflow/pkg/p2p/client.go:355\ngithub.com/pingcap/tiflow/pkg/p2p.(*MessageClient).run.func2\n\tgithub.com/pingcap/tiflow/pkg/p2p/client.go:205\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\tgolang.org/x/sync@v0.0.0-20220513210516-0976fa681c29/errgroup/errgroup.go:74\nruntime.goexit\n\truntime/asm_amd64.s:1571"]
[2022/08/05 15:03:35.484 +08:00] [INFO] [reactor_state.go:72] ["remote capture offline"] [captureID=c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9] [info="{\"id\":\"c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9\",\"address\":\"lingjin-4.jinling.tispace:8300\",\"version\":\"v6.2.0\"}"]
[2022/08/05 15:03:35.554 +08:00] [INFO] [reactor_state.go:72] ["remote capture offline"] [captureID=c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9] [info="{\"id\":\"c06ec3e8-7a8e-472c-ba6f-f0f8b21e3fc9\",\"address\":\"lingjin-4.jinling.tispace:8300\",\"version\":\"v6.2.0\"}"]
3AceShowHand commented 2 years ago

cdc-3-s.log cdc-4-s.log