search

tikv / migration

Migration tools for TiKV, e.g. online bulk load.
Apache License 2.0
35 stars 26 forks source link

cdc: clientSuite.TestStreamSendWithError is not stable #238

Closed pingyu closed 1 year ago

pingyu commented 2 years ago

Bug Report

1. Describe the bug

FAIL: client_test.go:1252: clientSuite.TestStreamSendWithError

client_test.go:1367:
    c.Assert(strings.Count(stack, "resolveLock"), check.Equals, 1)
... obtained int = 0
... expected int = 1

CI link:https://github.com/tikv/migration/actions/runs/3103756789/jobs/5027435144 logs (link):

cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/failpoint-ctl github.com/pingcap/failpoint/failpoint-ctl
go: downloading github.com/pingcap/failpoint v0.0.0-20210316064728-7acb0f0a3dfd
go: downloading github.com/sergi/go-diff v1.1.0
cd tools/check && GO111MODULE=on go build -mod=mod -o ../bin/gotestsum gotest.tools/gotestsum
go: downloading gotest.tools/gotestsum v1.8.1
go: downloading github.com/fatih/color v1.10.0
go: downloading github.com/dnephin/pflag v1.0.7
go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510
go: downloading github.com/pkg/errors v0.9.1
go: downloading golang.org/x/tools v0.1.1
go: downloading github.com/mattn/go-colorable v0.1.8
go: downloading github.com/mattn/go-isatty v0.0.14
go: downloading github.com/fsnotify/fsnotify v1.4.9
go: downloading golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e
go: downloading github.com/jonboulle/clockwork v0.2.2
go: downloading golang.org/x/sync v0.0.0-20210220032951-036812b2e83c
go: downloading golang.org/x/term v0.0.0-20210927222741-03fcf44c2211
go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1
go: downloading golang.org/x/mod v0.4.2
GO111MODULE=on go install github.com/axw/gocov/gocov@latest
go: downloading github.com/axw/gocov v1.1.0
go: downloading golang.org/x/tools v0.0.0-20190617190820-da514acc4774
GO111MODULE=on go install github.com/AlekSi/gocov-xml@latest
go: downloading github.com/AlekSi/gocov-xml v1.1.0
mkdir -p "/tmp/tikv_cdc_test.0YCTnJ9X"
$(echo $(for p in $(go list github.com/tikv/migration/cdc/... | grep -vE 'vendor|proto|cdc\/tests|integration|testing_utils|pb|pbmock'); do echo ${p#"github.com/tikv/migration/cdc/"}|grep -v "github.com/tikv/migration/cdc/"; done) | xargs tools/bin/failpoint-ctl enable >/dev/null)
go: downloading github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751
go: downloading github.com/swaggo/swag v1.8.3
go: downloading github.com/gin-gonic/gin v1.7.7
go: downloading github.com/google/uuid v1.3.0
go: downloading github.com/pingcap/errors v0.11.5-0.20211224045212-9687c2b0f87c
go: downloading github.com/pingcap/failpoint v0.0.0-20220423142525-ae43b7f4e5c3
go: downloading github.com/pingcap/log v1.1.0
go: downloading github.com/pingcap/tidb v1.1.0-beta.0.20220824050620-75c70ecd0aeb
go: downloading github.com/r3labs/diff v1.1.0
go: downloading github.com/tikv/client-go/v2 v2.0.1-0.20220818084834-0d0ae0dcfb1f
go: downloading github.com/tikv/pd/client v0.0.0-20220725055910-7187a7ab72db
go: downloading go.etcd.io/etcd/client/v3 v3.5.4
go: downloading github.com/tikv/pd v1.1.0-beta.0.20220725055910-7187a7ab72db
go: downloading go.etcd.io/etcd/server/v3 v3.5.4
go: downloading go.uber.org/zap v1.21.0
go: downloading golang.org/x/time v0.0.0-20220411224347-583f2d630306
go: downloading github.com/prometheus/client_golang v1.12.2
go: downloading github.com/swaggo/files v0.0.0-20210815190702-a29dd2bc99b2
go: downloading github.com/swaggo/gin-swagger v1.4.3
go: downloading go.etcd.io/etcd/client/pkg/v3 v3.5.4
go: downloading golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4
go: downloading google.golang.org/grpc v1.46.2
go: downloading github.com/tinylib/msgp v1.1.6
go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: downloading github.com/pingcap/kvproto v0.0.0-20220805093305-ab1ee4d521ab
go: downloading github.com/stretchr/testify v1.8.0
go: downloading github.com/edwingeng/deque v0.0.0-20191220032131-8596380dee17
go: downloading github.com/asaskevich/govalidator v0.0.0-20210307081110-f21760c49a8d
go: downloading github.com/twmb/murmur3 v1.1.3
go: downloading github.com/spf13/cobra v1.5.0
go: downloading github.com/fatih/color v1.13.0
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/BurntSushi/toml v1.2.0
go: downloading golang.org/x/net v0.0.0-20220722155237-a158d28d115b
go: downloading github.com/chzyer/readline v1.5.0
go: downloading github.com/mattn/go-shellwords v1.0.12
go: downloading github.com/pingcap/tidb-tools v6.0.1-0.20220516050036-b3ea358e374a+incompatible
go: downloading github.com/cockroachdb/pebble v0.0.0-20211124172904-3ca75111760c
go: downloading github.com/syndtr/goleveldb v1.0.1-0.20210305035536-64b5b1c73954
go: downloading github.com/benbjohnson/clock v1.1.0
go: downloading go.etcd.io/etcd/api/v3 v3.5.4
go: downloading github.com/Shopify/sarama v1.34.1
go: downloading go.uber.org/goleak v1.1.12
go: downloading github.com/pingcap/check v0.0.0-20211026125417-57bd13f7b5f0
go: downloading github.com/google/btree v1.1.2
go: downloading github.com/xdg/scram v1.0.5
go: downloading github.com/soheilhy/cmux v0.1.5
go: downloading go.uber.org/atomic v1.9.0
go: downloading github.com/coreos/go-semver v0.3.0
go: downloading github.com/golang/protobuf v1.5.2
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading github.com/bradleyjkemp/grpc-tools v0.2.7
go: downloading golang.org/x/exp v0.0.0-20220722155223-a9213eeb770e
go: downloading github.com/gin-contrib/sse v0.1.0
go: downloading github.com/KyleBanks/depth v1.2.1
go: downloading github.com/go-openapi/jsonreference v0.20.0
go: downloading github.com/go-openapi/spec v0.20.6
go: downloading golang.org/x/tools v0.1.12
go: downloading gopkg.in/natefinch/lumberjack.v2 v2.0.0
go: downloading github.com/coocood/freecache v1.2.1
go: downloading github.com/pingcap/tidb/parser v0.0.0-20220824050620-75c70ecd0aeb
go: downloading github.com/pingcap/tipb v0.0.0-20220718022156-3e2483c20a9e
go: downloading github.com/opentracing/opentracing-go v1.2.0
go: downloading go.etcd.io/etcd/pkg/v3 v3.5.4
go: downloading go.uber.org/multierr v1.8.0
go: downloading github.com/prometheus/client_model v0.2.0
go: downloading github.com/prometheus/common v0.32.1
go: downloading github.com/beorn7/perks v1.0.1
go: downloading github.com/cespare/xxhash/v2 v2.1.2
go: downloading github.com/prometheus/procfs v0.7.3
go: downloading google.golang.org/protobuf v1.28.0
go: downloading github.com/coreos/go-systemd/v22 v22.3.2
go: downloading github.com/philhofer/fwd v1.1.1
go: downloading github.com/uber/jaeger-client-go v2.22.1+incompatible
go: downloading google.golang.org/genproto v0.0.0-20220324131243-acbaeb5b85eb
go: downloading github.com/shirou/gopsutil/v3 v3.22.6
go: downloading github.com/mattn/go-colorable v0.1.12
go: downloading github.com/cockroachdb/errors v1.8.1
go: downloading github.com/cockroachdb/redact v1.0.8
go: downloading github.com/cakturk/go-netstat v0.0.0-20200220111822-e5b49efee7a5
go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0
go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20220101234140-673ab2c3ae75
go: downloading go.etcd.io/bbolt v1.3.6
go: downloading go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.20.0
go: downloading go.opentelemetry.io/contrib v0.20.0
go: downloading go.opentelemetry.io/otel/exporters/otlp v0.20.0
go: downloading go.opentelemetry.io/otel v0.20.0
go: downloading go.opentelemetry.io/otel/sdk v0.20.0
go: downloading golang.org/x/crypto v0.0.0-20220411220226-7b82a4e95df4
go: downloading sigs.k8s.io/yaml v1.2.0
go: downloading github.com/davecgh/go-spew v1.1.1
go: downloading github.com/eapache/go-resiliency v1.2.0
go: downloading github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21
go: downloading github.com/eapache/queue v1.1.0
go: downloading github.com/hashicorp/go-multierror v1.1.1
go: downloading github.com/jcmturner/gofork v1.0.0
go: downloading github.com/jcmturner/gokrb5/v8 v8.4.2
go: downloading github.com/klauspost/compress v1.15.6
go: downloading github.com/pierrec/lz4/v4 v4.1.14
go: downloading github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475
go: downloading github.com/xdg/stringprep v1.0.3
go: downloading github.com/improbable-eng/grpc-web v0.13.0
go: downloading github.com/sirupsen/logrus v1.9.0
go: downloading github.com/go-playground/validator/v10 v10.9.0
go: downloading github.com/ugorji/go/codec v1.2.6
go: downloading gopkg.in/yaml.v2 v2.4.0
go: downloading golang.org/x/sys v0.0.0-20220728004956-3c1f35247d10
go: downloading github.com/go-openapi/jsonpointer v0.19.5
go: downloading github.com/go-openapi/swag v0.21.1
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
go: downloading github.com/cznic/mathutil v0.0.0-20181122101859-297441e03548
go: downloading github.com/stathat/consistent v1.0.0
go: downloading github.com/dgryski/go-farm v0.0.0-20200201041132-a6ae2369ad13
go: downloading github.com/dustin/go-humanize v1.0.0
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading github.com/dgraph-io/ristretto v0.1.1-0.20220403145359-8e850b710d6d
go: downloading github.com/uber/jaeger-lib v2.4.1+incompatible
go: downloading github.com/pmezard/go-difflib v1.0.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/text v0.3.7
go: downloading github.com/cockroachdb/logtags v0.0.0-20190617123548-eb05cc24525f
go: downloading github.com/cockroachdb/sentry-go v0.6.1-cockroachdb.2
go: downloading github.com/DataDog/zstd v1.4.6-0.20210211175136-c6db21d202f4
go: downloading github.com/golang/snappy v0.0.4
go: downloading github.com/gorilla/websocket v1.4.2
go: downloading go.etcd.io/etcd/raft/v3 v3.5.4
go: downloading github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
go: downloading github.com/jonboulle/clockwork v0.3.0
go: downloading go.opentelemetry.io/otel/trace v0.20.0
go: downloading go.opentelemetry.io/otel/metric v0.20.0
go: downloading go.opentelemetry.io/otel/sdk/export/metric v0.20.0
go: downloading go.opentelemetry.io/otel/sdk/metric v0.20.0
go: downloading go.opentelemetry.io/proto/otlp v0.7.0
go: downloading github.com/hashicorp/errwrap v1.0.0
go: downloading github.com/jcmturner/dnsutils/v2 v2.0.0
go: downloading github.com/hashicorp/go-uuid v1.0.2
go: downloading github.com/desertbit/timer v0.0.0-20180107155436-c41aec40b27f
go: downloading github.com/rs/cors v1.7.0
go: downloading github.com/go-playground/universal-translator v0.18.0
go: downloading github.com/leodido/go-urn v1.2.1
go: downloading github.com/mailru/easyjson v0.7.7
go: downloading github.com/remyoudompheng/bigfft v0.0.0-20200410134404-eec4a21b6bb0
go: downloading github.com/gorilla/mux v1.8.0
go: downloading github.com/cheggaaa/pb/v3 v3.0.8
go: downloading github.com/ngaut/pools v0.0.0-20180318154953-b7bc8c42aac7
go: downloading github.com/kr/pretty v0.3.0
go: downloading github.com/form3tech-oss/jwt-go v3.2.5+incompatible
go: downloading go.etcd.io/etcd/client/v2 v2.305.4
go: downloading github.com/jcmturner/rpc/v2 v2.0.3
go: downloading github.com/danjacques/gofslock v0.0.0-20191023191349-0a45f885bc37
go: downloading github.com/opentracing/basictracer-go v1.0.0
go: downloading github.com/go-playground/locales v0.14.0
go: downloading github.com/josharian/intern v1.0.0
go: downloading github.com/golang/glog v1.0.0
go: downloading github.com/docker/go-units v0.4.0
go: downloading github.com/VividCortex/ewma v1.1.1
go: downloading github.com/mattn/go-runewidth v0.0.13
go: downloading github.com/pingcap/sysutil v0.0.0-20220114020952-ea68d2dbf5b4
go: downloading sourcegraph.com/sourcegraph/appdash v0.0.0-20190731080439-ebfcffb1b5c0
go: downloading github.com/iancoleman/strcase v0.2.0
go: downloading github.com/ngaut/sync2 v0.0.0-20141008032647-7a24ed77b2ef
go: downloading github.com/kr/text v0.2.0
go: downloading github.com/rogpeppe/go-internal v1.8.0
go: downloading github.com/jcmturner/aescts/v2 v2.0.0
go: downloading github.com/json-iterator/go v1.1.12
go: downloading github.com/modern-go/reflect2 v1.0.2
go: downloading cloud.google.com/go/storage v1.21.0
go: downloading github.com/Azure/azure-sdk-for-go/sdk/azidentity v0.12.0
go: downloading github.com/Azure/azure-sdk-for-go/sdk/storage/azblob v0.2.0
go: downloading cloud.google.com/go v0.100.2
go: downloading github.com/aliyun/alibaba-cloud-sdk-go v1.61.1581
go: downloading github.com/aws/aws-sdk-go v1.36.30
go: downloading golang.org/x/oauth2 v0.0.0-20220411215720-9780585627b5
go: downloading google.golang.org/api v0.74.0
go: downloading github.com/rivo/uniseg v0.2.0
go: downloading github.com/google/pprof v0.0.0-20220520215854-d04f2422c8a1
go: downloading github.com/wangjohn/quickselect v0.0.0-20161129230411-ed8402a42d5f
go: downloading golang.org/x/term v0.0.0-20220526004731-065cf7ba2467
go: downloading github.com/tklauser/go-sysconf v0.3.10
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading github.com/Azure/azure-sdk-for-go/sdk/azcore v0.20.0
go: downloading github.com/Azure/azure-sdk-for-go/sdk/internal v0.8.1
go: downloading github.com/pkg/browser v0.0.0-20180916011732-0a3d74bf9ce4
go: downloading cloud.google.com/go/compute v1.5.0
go: downloading cloud.google.com/go/iam v0.1.1
go: downloading github.com/googleapis/gax-go/v2 v2.2.0
go: downloading golang.org/x/xerrors v0.0.0-20220517211312-f3a8303e98df
go: downloading github.com/go-sql-driver/mysql v1.6.0
go: downloading github.com/tklauser/numcpus v0.4.0
go: downloading go.opencensus.io v0.23.0
go: downloading github.com/jmespath/go-jmespath v0.4.0
go: downloading github.com/pingcap/badger v1.5.1-0.20220314162537-ab58fbf40580
go: downloading github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989
go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da
go: downloading github.com/ncw/directio v1.0.5
go: downloading github.com/google/go-cmp v0.5.8
go: downloading github.com/coocood/rtutil v0.0.0-20190304133409-c84515f646f2
go: downloading github.com/coocood/bbloom v0.0.0-20190830030839-58deb6228d64
go: downloading github.com/klauspost/cpuid v1.3.1
go: downloading github.com/integralist/go-findroot v0.0.0-20160518114804-ac90681525dc
go: downloading github.com/phayes/freeport v0.0.0-20180830031419-95f893ade6f2
go: downloading github.com/stretchr/objx v0.4.0
∅  api
✓  cdc/capture (566ms) (coverage: 4.0% of statements)
✓  cdc/model (176ms) (coverage: 68.9% of statements)
✓  cdc/owner (3.306s) (coverage: 68.6% of statements)
✓  cdc (8.026s) (coverage: 47.8% of statements)
✓  cdc/processor (310ms) (coverage: 72.4% of statements)
✓  cdc/processor/pipeline (289ms) (coverage: 42.4% of statements)
✓  cdc/puller (336ms) (coverage: 84.0% of statements)
✓  cdc/scheduler (283ms) (coverage: 87.6% of statements)
✓  cdc/scheduler/util (129ms) (coverage: 90.3% of statements)
✓  cdc/sink (824ms) (coverage: 69.1% of statements)
∅  cdc/sink/common (24ms) (coverage: 0.0% of statements)
✓  cdc/puller/frontier (51.639s) (coverage: 92.7% of statements)
✓  cdc/sorter/memory (1m4.298s) (coverage: 65.8% of statements)
✓  cmd/cdc (509ms) (coverage: 100.0% of statements)
✓  pkg/api/internal/rest (2.14s) (coverage: 80.3% of statements)
✓  pkg/cmd/cli (492ms) (coverage: 23.6% of statements)
✓  pkg/cmd/server (589ms) (coverage: 62.5% of statements)
✓  pkg/cmd/util (164ms) (coverage: 58.7% of statements)
✓  pkg/config (82ms) (coverage: 71.7% of statements)
✓  pkg/context (1.23s) (coverage: 64.3% of statements)
✓  pkg/db (201ms) (coverage: 87.7% of statements)
✓  pkg/errors (25ms) (coverage: 92.9% of statements)
✓  pkg/etcd (10.744s) (coverage: 66.6% of statements)
✓  pkg/flags (22ms) (coverage: 100.0% of statements)
✓  pkg/fsutil (40ms) (coverage: 84.2% of statements)
✓  pkg/hash (35ms) (coverage: 90.0% of statements)
✓  pkg/httputil (122ms) (coverage: 90.0% of statements)
✓  pkg/leakutil (35ms) (coverage: 100.0% of statements)
✓  pkg/logutil (40ms) (coverage: 60.8% of statements)
✓  pkg/notify (5.724s) (coverage: 93.9% of statements)
✖  cdc/kv (3m44.069s) (coverage: 75.9% of statements)
∅  cdc/sorter
∅  cdc/sorter/encoding
✓  pkg/orchestrator/util (68ms) (coverage: 100.0% of statements)
✓  pkg/pdtime (1.58s) (coverage: 65.6% of statements)
✓  pkg/pipeline (1.561s) (coverage: 73.1% of statements)
✓  pkg/quotes (55ms) (coverage: 100.0% of statements)
✓  pkg/regionspan (317ms) (coverage: 80.1% of statements)
✓  pkg/retry (1.421s) (coverage: 98.0% of statements)
✓  pkg/scheduler (345ms) (coverage: 79.6% of statements)
✓  pkg/security (212ms) (coverage: 13.8% of statements)
✓  pkg/tcpserver (10.261s) (coverage: 87.0% of statements)
✓  pkg/txnutil/gc (2.897s) (coverage: 72.9% of statements)
✓  pkg/types (47ms) (coverage: 91.7% of statements)
✓  pkg/util (393ms) (coverage: 77.8% of statements)
✓  pkg/version (401ms) (coverage: 80.0% of statements)
✓  cdc/sorter/unified (3m49.604s) (coverage: 74.7% of statements)
∅  pkg/api/v1
∅  pkg/cmd
∅  pkg/cmd/context
∅  pkg/cmd/factory
∅  pkg/cmd/version
∅  pkg/config/outdated
∅  pkg/kafka
✓  pkg/orchestrator (2m35.18s) (coverage: 86.7% of statements)
∅  pkg/txnutil
∅  pkg/util/testleak
✓  pkg/workerpool (25.727s) (coverage: 92.4% of statements)

=== Failed
=== FAIL: cdc/kv Test (221.36s)
Error: 9/22 08:03:41.236 +00:00] [ERROR] [client.go:1041] ["tikv reported the request cluster ID mismatch error, which is not expected"] [tikvCurrentClusterID=0] [requestClusterID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1041\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75"]
Error: 9/22 08:03:42.119 +00:00] [ERROR] [region_worker.go:686] ["The CommitTs must be greater than the resolvedTs"] ["Event Type"=COMMIT] [CommitTs=98] [resolvedTs=100] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*regionWorker).handleEventEntry\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:686\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).processEvent\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:386\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).eventHandler\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:478\ngithub.com/tikv/migration/cdc/cdc/kv.(*regionWorker).run.func3\n\t/home/runner/work/migration/migration/cdc/cdc/kv/region_worker.go:601\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75"]
Error: 9/22 08:03:43.677 +00:00] [ERROR] [client.go:1036] ["tikv reported compatibility error, which is not expected"] [rpcCtx="region ID: 3, meta: id:3 region_epoch:<> peers:<id:4 store_id:1 > , peer: id:4 store_id:1 , addr: 127.0.0.1:39959, idx: 0, reqStoreType: TiKvOnly, runStoreType: tikv"] [error="required_version:\"v4.0.7\" "] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1036\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75"]
Error: 9/22 08:05:03.502 +00:00] [ERROR] [client.go:1032] ["tikv reported duplicated request to the same region, which is not expected"] [regionID=3] [stack="github.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).handleError\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:1032\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:574\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75"]
Error: 9/22 08:05:14.090 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 6.2.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 6.2.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:178\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75"]
Error: 9/22 08:05:14.192 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, the minimal compatible version is 6.2.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.1.0 is not supported, the minimal compatible version is 6.2.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:178\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75"]
Error: 9/22 08:05:14.294 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, the minimal compatible version is 6.2.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 2.1.10 is not supported, the minimal compatible version is 6.2.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:178\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75"]
Error: 9/22 08:05:14.395 +00:00] [ERROR] [client.go:365] ["check tikv version failed"] [error="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 6.2.0-alpha"] [errorVerbose="[CDC:ErrVersionIncompatible]version is incompatible: TiKV 3.0.10 is not supported, the minimal compatible version is 6.2.0-alpha\ngithub.com/pingcap/errors.AddStack\n\t/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/errors.go:174\ngithub.com/pingcap/errors.(*Error).GenWithStackByArgs\n\t/home/runner/go/pkg/mod/github.com/pingcap/errors@v0.11.5-0.20211224045212-9687c2b0f87c/normalize.go:164\ngithub.com/tikv/migration/cdc/pkg/version.CheckStoreVersion\n\t/home/runner/work/migration/migration/cdc/pkg/version/check.go:178\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:363\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.18.3/x64/src/runtime/asm_amd64.s:1571"] [storeID=1] [stack="github.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:365\ngithub.com/tikv/migration/cdc/pkg/retry.run\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:54\ngithub.com/tikv/migration/cdc/pkg/retry.Do\n\t/home/runner/work/migration/migration/cdc/pkg/retry/retry_with_opt.go:32\ngithub.com/tikv/migration/cdc/cdc/kv.(*CDCClient).newStream\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:351\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).requestRegionToStore\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:747\ngithub.com/tikv/migration/cdc/cdc/kv.(*eventFeedSession).eventFeed.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client.go:515\ngolang.org/x/sync/errgroup.(*Group).Go.func1\n\t/home/runner/go/pkg/mod/golang.org/x/sync@v0.0.0-20220722155255-886fb9371eb4/errgroup/errgroup.go:75"]
Error: 9/22 08:05:15.838 +00:00] [ERROR] [client_test.go:1710] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestIncompatibleTiKV.func2\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1710\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 9/22 08:06:21.712 +00:00] [ERROR] [client_test.go:2932] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).testKVClientForceReconnect.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:2932\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 9/22 08:06:23.412 +00:00] [ERROR] [client_test.go:2989] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).testKVClientForceReconnect.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:2989\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 9/22 08:06:26.233 +00:00] [ERROR] [client_test.go:3416] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3416\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 9/22 08:06:27.306 +00:00] [ERROR] [client_test.go:3502] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestPrewriteNotMatchError.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3502\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 9/22 08:06:40.290 +00:00] [ERROR] [client_test.go:3321] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestRegionWorkerExitWhenIsIdle.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3321\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 9/22 08:06:40.291 +00:00] [ERROR] [client_test.go:3321] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestRegionWorkerExitWhenIsIdle.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:3321\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 9/22 08:07:00.921 +00:00] [ERROR] [client_test.go:1505] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1505\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 9/22 08:07:00.961 +00:00] [ERROR] [client_test.go:1505] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamRecvWithErrorAndResolvedGoBack.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1505\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 9/22 08:07:06.051 +00:00] [ERROR] [client_test.go:1316] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamSendWithError.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1316\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 9/22 08:07:06.082 +00:00] [ERROR] [client_test.go:1316] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamSendWithError.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1316\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 9/22 08:07:06.116 +00:00] [ERROR] [client_test.go:1316] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamSendWithError.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1316\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]
Error: 9/22 08:07:06.187 +00:00] [ERROR] [client_test.go:1316] ["mock server error"] [error="rpc error: code = Canceled desc = context canceled"] [stack="github.com/tikv/migration/cdc/cdc/kv.(*clientSuite).TestStreamSendWithError.func4\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:1316\ngithub.com/tikv/migration/cdc/cdc/kv.(*mockChangeDataService).EventFeed.func1\n\t/home/runner/work/migration/migration/cdc/cdc/kv/client_test.go:253"]

----------------------------------------------------------------------
FAIL: client_test.go:1252: clientSuite.TestStreamSendWithError

client_test.go:1367:
    c.Assert(strings.Count(stack, "resolveLock"), check.Equals, 1)
... obtained int = 0
... expected int = 1

OOPS: 48 passed, 1 skipped, 1 FAILED

DONE 227 tests, 1 failure in 540.356s
make: *** [Makefile:119: unit_test_in_verify_ci] Error 1
Error: Process completed with exit code 2.

2. Minimal reproduce step (Required)

3. What did you see instead (Required)

4. What did you expect to see? (Required)

5. What is your migration tool and TiKV version? (Required)

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 30 days with no activity.