search

pingcap / tiflow

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)
Apache License 2.0
430 stars 286 forks source link

etcd test failed by data race #10771

Closed flowbehappy closed 8 months ago

flowbehappy commented 8 months ago

What did you do?

Run make cdc & make test

What did you expect to see?

Successfully finish.

What did you see instead?

)$ make cdc & make test
[1] 33885
CGO_ENABLED=1 GO111MODULE=on go build  -trimpath  -ldflags '-X "github.com/pingcap/tiflow/pkg/version.ReleaseVersion=v8.0.0-master" -X "github.com/pingcap/tiflow/pkg/version.BuildTS=2024-03-13 05:09:41" -X "github.com/pingcap/tiflow/pkg/version.GitHash=df8c65ea59612c639b8ccce1d576cd63b1b173b8" -X "github.com/pingcap/tiflow/pkg/version.GitBranch=master" -X "github.com/pingcap/tiflow/pkg/version.GoVersion=go version go1.21.6 darwin/amd64" -X "github.com/pingcap/tidb/pkg/parser/mysql.TiDBReleaseVersion=v8.0.0-master"' -o bin/cdc ./cmd/cdc
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl
scripts/generate-mock.sh
go generate
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/protoc-gen-gogofaster github.com/gogo/protobuf/protoc-gen-gogofaster
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/protoc-gen-go google.golang.org/protobuf/cmd/protoc-gen-go
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/protoc-gen-go-grpc google.golang.org/grpc/cmd/protoc-gen-go-grpc
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/protoc-gen-grpc-gateway github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/protoc-gen-grpc-gateway-v2 github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/protoc-gen-openapiv2 github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2
generate-protobuf
./scripts/generate-protobuf.sh
generate ./proto/EntryProtocol.proto...
generate ./proto/CanalProtocol.proto...
generate ./proto/CraftBenchmark.proto...
generate ./proto/CDCPeerToPeer.proto...
generate ./dm/proto/dmworker.proto...
generate ./dm/proto/dmmaster.proto...
generate cdc/processor/tablepb/table.proto...
generate cdc/scheduler/schedulepb/table_schedule.proto...
generate enginepb...
mkdir -p "/tmp/tidb_cdc_test"
$(echo $(for p in $(go list ./... | grep -vE 'vendor|proto|tiflow/tests|integration|testing_utils|pb|pbmock|tiflow/bin'); do echo ${p#"github.com/pingcap/tiflow/"}|grep -v "github.com/pingcap/tiflow"; done) | xargs tools/bin/failpoint-ctl enable >/dev/null)
ok      github.com/pingcap/tiflow/cdc   5.135s  coverage: 95.5% of statements
ok      github.com/pingcap/tiflow/cdc/api   4.471s  coverage: 8.8% of statements
?       github.com/pingcap/tiflow/cdc/api/status    [no test files]
ok      github.com/pingcap/tiflow/cdc/api/middleware    6.071s  coverage: 20.0% of statements
ok      github.com/pingcap/tiflow/cdc/api/owner 4.400s  coverage: 22.8% of statements
ok      github.com/pingcap/tiflow/cdc/api/v1    4.855s  coverage: 56.4% of statements
?       github.com/pingcap/tiflow/cdc/capture/mock  [no test files]
?       github.com/pingcap/tiflow/cdc/controller/mock   [no test files]
ok      github.com/pingcap/tiflow/cdc/api/v2    24.277s coverage: 70.2% of statements
ok      github.com/pingcap/tiflow/cdc/capture   5.591s  coverage: 20.7% of statements
ok      github.com/pingcap/tiflow/cdc/controller    4.605s  coverage: 43.7% of statements
?       github.com/pingcap/tiflow/cdc/owner/mock    [no test files]
ok      github.com/pingcap/tiflow/cdc/entry 100.544s    coverage: 61.2% of statements
ok      github.com/pingcap/tiflow/cdc/entry/schema  4.402s  coverage: 58.0% of statements
ok      github.com/pingcap/tiflow/cdc/kv    82.931s coverage: 74.3% of statements
ok      github.com/pingcap/tiflow/cdc/kv/regionlock 3.505s  coverage: 91.6% of statements
ok      github.com/pingcap/tiflow/cdc/kv/sharedconn 5.674s  coverage: 84.9% of statements
ok      github.com/pingcap/tiflow/cdc/metrics   1.432s  coverage: 100.0% of statements
ok      github.com/pingcap/tiflow/cdc/model 3.001s  coverage: 55.4% of statements
ok      github.com/pingcap/tiflow/cdc/model/codec   2.865s  coverage: 92.1% of statements
ok      github.com/pingcap/tiflow/cdc/model/codec/v1    2.890s  coverage: 53.2% of statements
?       github.com/pingcap/tiflow/cdc/processor/mock    [no test files]
?       github.com/pingcap/tiflow/cdc/processor/sourcemanager   [no test files]
?       github.com/pingcap/tiflow/cdc/processor/sourcemanager/puller    [no test files]
ok      github.com/pingcap/tiflow/cdc/owner 48.509s coverage: 73.6% of statements
ok      github.com/pingcap/tiflow/cdc/processor 6.017s  coverage: 54.6% of statements
ok      github.com/pingcap/tiflow/cdc/processor/memquota    2.991s  coverage: 87.0% of statements
ok      github.com/pingcap/tiflow/cdc/processor/sinkmanager 5.891s  coverage: 73.2% of statements
?       github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter/factory    [no test files]
ok      github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter    4.234s  coverage: 50.0% of statements
?       github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter/mock   [no test files]
ok      github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter/memory 4.371s  coverage: 62.7% of statements
ok      github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter/pebble 6.532s  coverage: 88.9% of statements
ok      github.com/pingcap/tiflow/cdc/processor/sourcemanager/sorter/pebble/encoding    2.913s  coverage: 69.1% of statements
ok      github.com/pingcap/tiflow/cdc/puller    18.244s coverage: 64.1% of statements
ok      github.com/pingcap/tiflow/cdc/puller/frontier   56.978s coverage: 96.1% of statements
?       github.com/pingcap/tiflow/cdc/redo/writer   [no test files]
?       github.com/pingcap/tiflow/cdc/redo/writer/blackhole [no test files]
?       github.com/pingcap/tiflow/cdc/redo/writer/factory   [no test files]
ok      github.com/pingcap/tiflow/cdc/puller/memorysorter   60.829s coverage: 64.2% of statements
ok      github.com/pingcap/tiflow/cdc/redo  10.743s coverage: 77.3% of statements
ok      github.com/pingcap/tiflow/cdc/redo/common   2.826s  coverage: 56.0% of statements
ok      github.com/pingcap/tiflow/cdc/redo/reader   6.647s  coverage: 71.1% of statements
?       github.com/pingcap/tiflow/cdc/scheduler [no test files]
?       github.com/pingcap/tiflow/cdc/scheduler/internal    [no test files]
ok      github.com/pingcap/tiflow/cdc/redo/writer/file  3.136s  coverage: 72.9% of statements
ok      github.com/pingcap/tiflow/cdc/redo/writer/memory    2.875s  coverage: 79.6% of statements
ok      github.com/pingcap/tiflow/cdc/scheduler/internal/v3 6.420s  coverage: 67.8% of statements
ok      github.com/pingcap/tiflow/cdc/scheduler/internal/v3/agent   5.094s  coverage: 86.3% of statements
ok      github.com/pingcap/tiflow/cdc/scheduler/internal/v3/compat  2.973s  coverage: 92.4% of statements
ok      github.com/pingcap/tiflow/cdc/scheduler/internal/v3/keyspan 8.573s  coverage: 89.3% of statements
ok      github.com/pingcap/tiflow/cdc/scheduler/internal/v3/member  3.233s  coverage: 90.0% of statements
ok      github.com/pingcap/tiflow/cdc/scheduler/internal/v3/replication 10.041s coverage: 72.6% of statements
ok      github.com/pingcap/tiflow/cdc/scheduler/internal/v3/scheduler   3.095s  coverage: 90.7% of statements
ok      github.com/pingcap/tiflow/cdc/scheduler/internal/v3/transport   2.895s  coverage: 48.2% of statements
?       github.com/pingcap/tiflow/cdc/sink/ddlsink  [no test files]
?       github.com/pingcap/tiflow/cdc/sink/ddlsink/blackhole    [no test files]
?       github.com/pingcap/tiflow/cdc/sink/ddlsink/factory  [no test files]
ok      github.com/pingcap/tiflow/cdc/server    21.685s coverage: 40.6% of statements
ok      github.com/pingcap/tiflow/cdc/sink/ddlsink/cloudstorage 9.022s  coverage: 60.2% of statements
ok      github.com/pingcap/tiflow/cdc/sink/ddlsink/mq   4.663s  coverage: 74.0% of statements
ok      github.com/pingcap/tiflow/cdc/sink/ddlsink/mq/ddlproducer   2.877s  coverage: 21.1% of statements
ok      github.com/pingcap/tiflow/cdc/sink/ddlsink/mysql    7.955s  coverage: 70.0% of statements
ok      github.com/pingcap/tiflow/cdc/sink/dmlsink  2.888s  coverage: 92.3% of statements
ok      github.com/pingcap/tiflow/cdc/sink/dmlsink/blackhole    3.769s  coverage: 75.0% of statements
ok      github.com/pingcap/tiflow/cdc/sink/dmlsink/cloudstorage 16.588s coverage: 78.4% of statements
ok      github.com/pingcap/tiflow/cdc/sink/dmlsink/factory  4.934s  coverage: 12.2% of statements
ok      github.com/pingcap/tiflow/cdc/sink/dmlsink/mq   23.015s coverage: 64.1% of statements
ok      github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dispatcher    2.925s  coverage: 78.9% of statements
ok      github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dispatcher/partition  3.320s  coverage: 89.5% of statements
ok      github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dispatcher/topic  1.498s  coverage: 76.5% of statements
?       github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/transformer   [no test files]
ok      github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/dmlproducer   3.064s  coverage: 23.4% of statements
ok      github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/manager   3.591s  coverage: 61.4% of statements
?       github.com/pingcap/tiflow/cdc/sink/metrics  [no test files]
?       github.com/pingcap/tiflow/cdc/sink/metrics/cloudstorage [no test files]
?       github.com/pingcap/tiflow/cdc/sink/metrics/mq   [no test files]
?       github.com/pingcap/tiflow/cdc/sink/metrics/tablesink    [no test files]
?       github.com/pingcap/tiflow/cdc/sink/metrics/txn  [no test files]
ok      github.com/pingcap/tiflow/cdc/sink/dmlsink/mq/transformer/columnselector    10.980s coverage: 84.6% of statements
ok      github.com/pingcap/tiflow/cdc/sink/dmlsink/txn  8.007s  coverage: 75.4% of statements
ok      github.com/pingcap/tiflow/cdc/sink/dmlsink/txn/mysql    11.377s coverage: 84.7% of statements
ok      github.com/pingcap/tiflow/cdc/sink/tablesink    3.608s  coverage: 91.6% of statements
ok      github.com/pingcap/tiflow/cdc/sink/tablesink/state  1.468s  coverage: 16.7% of statements
ok      github.com/pingcap/tiflow/cdc/sink/util 3.010s  coverage: 22.6% of statements
?       github.com/pingcap/tiflow/cdc/syncpointstore    [no test files]
?       github.com/pingcap/tiflow/cdcv2/capture [no test files]
?       github.com/pingcap/tiflow/cdcv2/controller  [no test files]
ok      github.com/pingcap/tiflow/cdc/sink/validator    4.882s  coverage: 58.8% of statements
?       github.com/pingcap/tiflow/cdcv2/owner   [no test files]
?       github.com/pingcap/tiflow/docs/swagger  [no test files]
ok      github.com/pingcap/tiflow/cdcv2/metadata    4.506s  coverage: 0.0% of statements [no tests to run]
ok      github.com/pingcap/tiflow/cdcv2/metadata/sql    5.907s  coverage: 32.8% of statements
?       github.com/pingcap/tiflow/pkg/api/v2    [no test files]
?       github.com/pingcap/tiflow/pkg/api/v2/mock   [no test files]
ok      github.com/pingcap/tiflow/pkg/api/internal/rest 6.719s  coverage: 80.5% of statements
?       github.com/pingcap/tiflow/pkg/causality [no test files]
ok      github.com/pingcap/tiflow/pkg/applier   5.272s  coverage: 72.7% of statements
ok      github.com/pingcap/tiflow/pkg/causality/internal    2.445s  coverage: 94.3% of statements
ok      github.com/pingcap/tiflow/pkg/causality/tests   3.527s  coverage: 93.9% of statements
?       github.com/pingcap/tiflow/pkg/cmd   [no test files]
ok      github.com/pingcap/tiflow/pkg/chann 13.613s coverage: 88.2% of statements
ok      github.com/pingcap/tiflow/pkg/chdelay   3.489s  coverage: 97.9% of statements
?       github.com/pingcap/tiflow/pkg/cmd/context   [no test files]
ok      github.com/pingcap/tiflow/pkg/cmd/cli   4.531s  coverage: 68.8% of statements
?       github.com/pingcap/tiflow/pkg/cmd/factory/mock  [no test files]
?       github.com/pingcap/tiflow/pkg/cmd/version   [no test files]
?       github.com/pingcap/tiflow/pkg/compression   [no test files]
ok      github.com/pingcap/tiflow/pkg/cmd/factory   22.711s coverage: 10.9% of statements
ok      github.com/pingcap/tiflow/pkg/cmd/redo  4.523s  coverage: 14.8% of statements
ok      github.com/pingcap/tiflow/pkg/cmd/server    5.603s  coverage: 58.3% of statements
ok      github.com/pingcap/tiflow/pkg/cmd/util  3.013s  coverage: 73.3% of statements
?       github.com/pingcap/tiflow/pkg/config/outdated   [no test files]
ok      github.com/pingcap/tiflow/pkg/config    3.442s  coverage: 69.4% of statements
ok      github.com/pingcap/tiflow/pkg/container/queue   3.760s  coverage: 98.1% of statements
ok      github.com/pingcap/tiflow/pkg/container/sortmap 2.552s  coverage: 100.0% of statements
?       github.com/pingcap/tiflow/pkg/election/mock [no test files]
ok      github.com/pingcap/tiflow/pkg/context   5.456s  coverage: 69.2% of statements
ok      github.com/pingcap/tiflow/pkg/election  11.624s coverage: 87.2% of statements
ok      github.com/pingcap/tiflow/pkg/errors    2.446s  coverage: 95.1% of statements
?       github.com/pingcap/tiflow/pkg/etcd/mock [no test files]
ok      github.com/pingcap/tiflow/pkg/errorutil 3.592s  coverage: 88.0% of statements
[2024/03/13 13:38:27.991 +08:00] [WARN] [client.go:286] ["etcd client watchCh blocking too long, reset the watchCh"] [duration=10s] [stack="github.com/pingcap/tiflow/pkg/etcd.(*Client).WatchWithChan\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:288\ngithub.com/pingcap/tiflow/pkg/etcd.TestRevisionNotFallBack.func2\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:302"] [role=]
[2024/03/13 13:38:28.002 +08:00] [WARN] [client.go:286] ["etcd client watchCh blocking too long, reset the watchCh"] [duration=10s] [stack="github.com/pingcap/tiflow/pkg/etcd.(*Client).WatchWithChan\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:288\ngithub.com/pingcap/tiflow/pkg/etcd.TestRevisionNotFallBack.func2\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:302"] [role=]
[2024/03/13 13:38:28.014 +08:00] [WARN] [client.go:286] ["etcd client watchCh blocking too long, reset the watchCh"] [duration=10s] [stack="github.com/pingcap/tiflow/pkg/etcd.(*Client).WatchWithChan\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:288\ngithub.com/pingcap/tiflow/pkg/etcd.TestRevisionNotFallBack.func2\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:302"] [role=]
[2024/03/13 13:38:28.017 +08:00] [INFO] [client.go:248] ["WatchWithChan exited"] [role=]
[2024/03/13 13:38:28.937 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Revoke] [error="etcdserver: requested lease not found"]
[2024/03/13 13:38:29.050 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=10s] [role=]
[2024/03/13 13:38:29.051 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=11s] [role=]
[2024/03/13 13:38:29.052 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=12s] [role=]
[2024/03/13 13:38:29.054 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=13s] [role=]
[2024/03/13 13:38:29.055 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=14s] [role=]
[2024/03/13 13:38:29.056 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=15s] [role=]
[2024/03/13 13:38:29.057 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=16s] [role=]
[2024/03/13 13:38:29.059 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=17s] [role=]
[2024/03/13 13:38:29.061 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=18s] [role=]
[2024/03/13 13:38:29.062 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=19s] [role=]
[2024/03/13 13:38:29.063 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=20s] [role=]
[2024/03/13 13:38:29.064 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=21s] [role=]
[2024/03/13 13:38:29.065 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=22s] [role=]
[2024/03/13 13:38:29.066 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=23s] [role=]
[2024/03/13 13:38:29.068 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=24s] [role=]
[2024/03/13 13:38:29.069 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=25s] [role=]
[2024/03/13 13:38:29.070 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=26s] [role=]
[2024/03/13 13:38:29.071 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=27s] [role=]
[2024/03/13 13:38:29.072 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=28s] [role=]
[2024/03/13 13:38:29.075 +08:00] [WARN] [client.go:272] ["etcd client outCh blocking too long, the etcdWorker may be stuck"] [duration=29s] [role=]
[2024/03/13 13:38:29.077 +08:00] [INFO] [client.go:248] ["WatchWithChan exited"] [role=]
[2024/03/13 13:38:29.078 +08:00] [PANIC] [etcdkey.go:219] [unreachable] [stack="github.com/pingcap/tiflow/pkg/etcd.(*CDCKey).String\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/etcdkey.go:219\ngithub.com/pingcap/tiflow/pkg/etcd.TestEtcdKeyParseError.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/etcdkey_test.go:169\ngithub.com/stretchr/testify/assert.didPanic\n\t/Users/flow/workspace/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/assert/assertions.go:1158\ngithub.com/stretchr/testify/assert.Panics\n\t/Users/flow/workspace/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/assert/assertions.go:1172\ngithub.com/stretchr/testify/require.Panics\n\t/Users/flow/workspace/gopath/pkg/mod/github.com/stretchr/testify@v1.8.4/require/require.go:1718\ngithub.com/pingcap/tiflow/pkg/etcd.TestEtcdKeyParseError\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/etcdkey_test.go:168\ntesting.tRunner\n\t/usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595"]
==================
WARNING: DATA RACE
Write at 0x000108c45ca0 by goroutine 152:
  github.com/pingcap/tiflow/pkg/etcd.TestRetry()
      /Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:78 +0x64
  testing.tRunner()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x44

Previous read at 0x000108c45ca0 by goroutine 168:
  github.com/pingcap/tiflow/pkg/etcd.retryRPC()
      /Users/flow/workspace/github/tiflow/pkg/etcd/client.go:118 +0x109
  github.com/pingcap/tiflow/pkg/etcd.(*Client).Put()
      /Users/flow/workspace/github/tiflow/pkg/etcd/client.go:128 +0x2da
  github.com/pingcap/tiflow/pkg/etcd.TestDeleteCaptureInfo()
      /Users/flow/workspace/github/tiflow/pkg/etcd/etcd_test.go:462 +0x85b
  testing.tRunner()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x44

Goroutine 152 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x845
  testing.runTests.func1()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:2054 +0x84
  testing.tRunner()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x261
  testing.runTests()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:2052 +0x8ad
  testing.(*M).Run()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1925 +0xcd7
  go.uber.org/goleak.VerifyTestMain()
      /Users/flow/workspace/gopath/pkg/mod/go.uber.org/goleak@v1.3.0/testmain.go:53 +0x64
  github.com/pingcap/tiflow/pkg/leakutil.SetUpLeakTest()
      /Users/flow/workspace/github/tiflow/pkg/leakutil/leak_helper.go:48 +0x48a
  github.com/pingcap/tiflow/pkg/etcd.TestMain()
      /Users/flow/workspace/github/tiflow/pkg/etcd/main_test.go:23 +0x329
  main.main()
      _testmain.go:121 +0x328

Goroutine 168 (running) created at:
  testing.(*T).Run()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1648 +0x845
  testing.runTests.func1()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:2054 +0x84
  testing.tRunner()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595 +0x261
  testing.runTests()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:2052 +0x8ad
  testing.(*M).Run()
      /usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1925 +0xcd7
  go.uber.org/goleak.VerifyTestMain()
      /Users/flow/workspace/gopath/pkg/mod/go.uber.org/goleak@v1.3.0/testmain.go:53 +0x64
  github.com/pingcap/tiflow/pkg/leakutil.SetUpLeakTest()
      /Users/flow/workspace/github/tiflow/pkg/leakutil/leak_helper.go:48 +0x48a
  github.com/pingcap/tiflow/pkg/etcd.TestMain()
      /Users/flow/workspace/github/tiflow/pkg/etcd/main_test.go:23 +0x329
  main.main()
      _testmain.go:121 +0x328
==================
[2024/03/13 13:38:30.001 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Put] [error="mock error"] [errorVerbose="mock error\ngithub.com/pingcap/tiflow/pkg/etcd.(*mockClient).Put\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:43\ngithub.com/pingcap/tiflow/pkg/etcd.(*Client).Put.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:130\ngithub.com/pingcap/tiflow/pkg/etcd.retryRPC.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:108\ngithub.com/pingcap/tiflow/pkg/retry.run\n\t/Users/flow/workspace/github/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\t/Users/flow/workspace/github/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/pkg/etcd.retryRPC\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:107\ngithub.com/pingcap/tiflow/pkg/etcd.(*Client).Put\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:128\ngithub.com/pingcap/tiflow/pkg/etcd.TestRetry\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:88\ntesting.tRunner\n\t/usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595\nruntime.goexit\n\t/usr/local/Cellar/go/1.21.6/libexec/src/runtime/asm_amd64.s:1650"]
[2024/03/13 13:38:30.007 +08:00] [WARN] [client.go:286] ["etcd client watchCh blocking too long, reset the watchCh"] [duration=10s] [stack="github.com/pingcap/tiflow/pkg/etcd.(*Client).WatchWithChan\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:288\ngithub.com/pingcap/tiflow/pkg/etcd.TestWatchChBlocked.func2\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:191"] [role=]
[2024/03/13 13:38:30.018 +08:00] [WARN] [client.go:286] ["etcd client watchCh blocking too long, reset the watchCh"] [duration=10s] [stack="github.com/pingcap/tiflow/pkg/etcd.(*Client).WatchWithChan\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:288\ngithub.com/pingcap/tiflow/pkg/etcd.TestWatchChBlocked.func2\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:191"] [role=]
[2024/03/13 13:38:30.030 +08:00] [INFO] [client.go:248] ["WatchWithChan exited"] [role=]
--- FAIL: TestWatchChBlocked (0.04s)
    testing.go:1465: race detected during execution of test
--- FAIL: TestCheckMultipleCDCClusterExist (2.61s)
    testing.go:61: <nil>
    testing.go:1465: race detected during execution of test
--- FAIL: TestDeleteCaptureInfo (1.61s)
    testing.go:61: <nil>
    testing.go:1465: race detected during execution of test
--- FAIL: TestGetAllChangeFeedInfo (1.28s)
    testing.go:61: <nil>
    testing.go:1465: race detected during execution of test
[2024/03/13 13:38:31.591 +08:00] [WARN] [etcd.go:522] ["unexpected etcd transaction failure, operation: Create"] [namespace=test] [changefeed=create-changefeed]
[2024/03/13 13:38:32.299 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Put] [error="mock error"] [errorVerbose="mock error\ngithub.com/pingcap/tiflow/pkg/etcd.(*mockClient).Put\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:43\ngithub.com/pingcap/tiflow/pkg/etcd.(*Client).Put.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:130\ngithub.com/pingcap/tiflow/pkg/etcd.retryRPC.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:108\ngithub.com/pingcap/tiflow/pkg/retry.run\n\t/Users/flow/workspace/github/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\t/Users/flow/workspace/github/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/pkg/etcd.retryRPC\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:107\ngithub.com/pingcap/tiflow/pkg/etcd.(*Client).Put\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:128\ngithub.com/pingcap/tiflow/pkg/etcd.TestRetry\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:88\ntesting.tRunner\n\t/usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595\nruntime.goexit\n\t/usr/local/Cellar/go/1.21.6/libexec/src/runtime/asm_amd64.s:1650"]
[2024/03/13 13:38:32.300 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Txn] [error="etcdserver: mvcc: database space exceeded"]
--- FAIL: TestGetOwnerRevision (2.35s)
    testing.go:61: <nil>
    testing.go:1465: race detected during execution of test
[2024/03/13 13:38:34.324 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Txn] [error="etcdserver: mvcc: database space exceeded"]
[2024/03/13 13:38:34.324 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Txn] [error="etcdserver: request timed out, possibly due to previous leader failure"]
[2024/03/13 13:38:36.527 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Txn] [error="etcdserver: request timed out, possibly due to previous leader failure"]
[2024/03/13 13:38:36.528 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Txn] [error="context deadline exceeded"]
[2024/03/13 13:38:36.528 +08:00] [WARN] [client.go:110] ["etcd RPC failed"] [RPC=Txn] [error="mock error"] [errorVerbose="mock error\ngithub.com/pingcap/tiflow/pkg/etcd.(*mockTxn).Commit\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:353\ngithub.com/pingcap/tiflow/pkg/etcd.(*Client).Txn.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:172\ngithub.com/pingcap/tiflow/pkg/etcd.retryRPC.func1\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:108\ngithub.com/pingcap/tiflow/pkg/retry.run\n\t/Users/flow/workspace/github/tiflow/pkg/retry/retry_with_opt.go:57\ngithub.com/pingcap/tiflow/pkg/retry.Do\n\t/Users/flow/workspace/github/tiflow/pkg/retry/retry_with_opt.go:34\ngithub.com/pingcap/tiflow/pkg/etcd.retryRPC\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:107\ngithub.com/pingcap/tiflow/pkg/etcd.(*Client).Txn\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client.go:170\ngithub.com/pingcap/tiflow/pkg/etcd.TestRetry\n\t/Users/flow/workspace/github/tiflow/pkg/etcd/client_test.go:114\ntesting.tRunner\n\t/usr/local/Cellar/go/1.21.6/libexec/src/testing/testing.go:1595\nruntime.goexit\n\t/usr/local/Cellar/go/1.21.6/libexec/src/runtime/asm_amd64.s:1650"]
--- FAIL: TestRetry (6.53s)
    testing.go:1465: race detected during execution of test
FAIL
coverage: 62.9% of statements
FAIL    github.com/pingcap/tiflow/pkg/etcd  11.173s
?       github.com/pingcap/tiflow/pkg/integrity [no test files]
ok      github.com/pingcap/tiflow/pkg/filter    18.301s coverage: 89.0% of statements
ok      github.com/pingcap/tiflow/pkg/flags 1.460s  coverage: 100.0% of statements
ok      github.com/pingcap/tiflow/pkg/fsutil    2.865s  coverage: 89.0% of statements
ok      github.com/pingcap/tiflow/pkg/hash  1.403s  coverage: 90.0% of statements
ok      github.com/pingcap/tiflow/pkg/httputil  2.491s  coverage: 60.0% of statements
ok      github.com/pingcap/tiflow/pkg/label 1.431s  coverage: 88.2% of statements
ok      github.com/pingcap/tiflow/pkg/leakutil  1.414s  coverage: 50.0% of statements
ok      github.com/pingcap/tiflow/pkg/logutil   4.628s  coverage: 56.2% of statements
ok      github.com/pingcap/tiflow/pkg/migrate   12.099s coverage: 77.2% of statements
ok      github.com/pingcap/tiflow/pkg/notify    6.846s  coverage: 100.0% of statements
?       github.com/pingcap/tiflow/pkg/sink  [no test files]
ok      github.com/pingcap/tiflow/pkg/orchestrator  186.972s    coverage: 69.0% of statements
ok      github.com/pingcap/tiflow/pkg/orchestrator/util 1.403s  coverage: 100.0% of statements
ok      github.com/pingcap/tiflow/pkg/p2p   14.233s coverage: 82.8% of statements
ok      github.com/pingcap/tiflow/pkg/p2p/internal  3.055s  coverage: 85.0% of statements
ok      github.com/pingcap/tiflow/pkg/pdutil    37.864s coverage: 69.4% of statements
ok      github.com/pingcap/tiflow/pkg/quotes    1.467s  coverage: 100.0% of statements
ok      github.com/pingcap/tiflow/pkg/redo  2.644s  coverage: 37.9% of statements
ok      github.com/pingcap/tiflow/pkg/retry 4.666s  coverage: 94.4% of statements
ok      github.com/pingcap/tiflow/pkg/security  1.692s  coverage: 12.1% of statements
ok      github.com/pingcap/tiflow/pkg/sink/cloudstorage 3.132s  coverage: 79.4% of statements
ok      github.com/pingcap/tiflow/pkg/sink/codec    7.267s  coverage: 40.5% of statements
ok      github.com/pingcap/tiflow/pkg/sink/codec/avro   6.855s  coverage: 71.5% of statements
ok      github.com/pingcap/tiflow/pkg/sink/codec/builder    4.208s  coverage: 0.0% of statements
?       github.com/pingcap/tiflow/pkg/sink/codec/internal   [no test files]
?       github.com/pingcap/tiflow/pkg/sink/codec/utils  [no test files]
?       github.com/pingcap/tiflow/pkg/sink/kafka/v2/mock    [no test files]
ok      github.com/pingcap/tiflow/pkg/sink/codec/canal  112.177s    coverage: 75.0% of statements
ok      github.com/pingcap/tiflow/pkg/sink/codec/common 2.928s  coverage: 38.4% of statements
ok      github.com/pingcap/tiflow/pkg/sink/codec/craft  4.629s  coverage: 84.5% of statements
ok      github.com/pingcap/tiflow/pkg/sink/codec/csv    3.188s  coverage: 80.7% of statements
ok      github.com/pingcap/tiflow/pkg/sink/codec/debezium   11.911s coverage: 92.7% of statements
ok      github.com/pingcap/tiflow/pkg/sink/codec/maxwell    3.315s  coverage: 50.7% of statements
ok      github.com/pingcap/tiflow/pkg/sink/codec/open   4.385s  coverage: 71.0% of statements
ok      github.com/pingcap/tiflow/pkg/sink/codec/simple 66.909s coverage: 79.9% of statements
ok      github.com/pingcap/tiflow/pkg/sink/kafka    3.717s  coverage: 58.4% of statements
ok      github.com/pingcap/tiflow/pkg/sink/kafka/claimcheck 3.482s  coverage: 28.0% of statements
ok      github.com/pingcap/tiflow/pkg/sink/kafka/v2 3.946s  coverage: 76.2% of statements
ok      github.com/pingcap/tiflow/pkg/sink/mysql    5.547s  coverage: 60.5% of statements
ok      github.com/pingcap/tiflow/pkg/sink/observer 5.110s  coverage: 77.0% of statements
ok      github.com/pingcap/tiflow/pkg/sink/pulsar   2.935s  coverage: 39.0% of statements
ok      github.com/pingcap/tiflow/pkg/spanz 6.366s  coverage: 61.1% of statements
?       github.com/pingcap/tiflow/pkg/txnutil   [no test files]
ok      github.com/pingcap/tiflow/pkg/sqlmodel  3.786s  coverage: 78.7% of statements
ok      github.com/pingcap/tiflow/pkg/tcpserver 13.071s coverage: 87.0% of statements
ok      github.com/pingcap/tiflow/pkg/txnutil/gc    5.838s  coverage: 70.8% of statements
ok      github.com/pingcap/tiflow/pkg/types 1.562s  coverage: 91.7% of statements
ok      github.com/pingcap/tiflow/pkg/upstream  6.757s  coverage: 60.7% of statements
ok      github.com/pingcap/tiflow/pkg/util  3.100s  coverage: 48.9% of statements
ok      github.com/pingcap/tiflow/pkg/uuid  1.404s  coverage: 100.0% of statements
ok      github.com/pingcap/tiflow/pkg/version   23.248s coverage: 85.1% of statements
ok      github.com/pingcap/tiflow/pkg/workerpool    26.955s coverage: 86.8% of statements
FAIL
make: *** [unit_test] Error 1
[1]+  Done                    make cdc

Versions of the cluster

Master

fubinzh commented 8 months ago

/severity moderate

fubinzh commented 8 months ago

/severity moderate