search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.01k stars 3.79k forks source link

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #113923

Closed cockroach-teamcity closed 11 months ago

cockroach-teamcity commented 11 months ago

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed on master @ 414cb46513079bb8869bd776f045ed6c96ad6d26:

                              | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*keyLocks).enqueueLockingRequest
                              |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:2733
                              | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*keyLocks).scanAndMaybeEnqueue
                              |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:2386
                              | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableGuardImpl).resumeScan
                              |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:931
                              | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).ScanAndEnqueue
                              |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:3855
                              | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard
                              |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:335
                              | github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq
                              |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:239
                              | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries
                              |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468
                              | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes
                              |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185
                              | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes
                              |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193
                              | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes
                              |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202
                              | github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal
                              |     github.com/cockroachdb/cockroach/pkg/server/node.go:1333
                              | github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch
                              |     github.com/cockroachdb/cockroach/pkg/server/node.go:1464
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704
                              | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
                              |     github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
                              | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
                              | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
                              | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
                            Wraps: (2) secondary error attachment
                              | runtime error: invalid memory address or nil pointer dereference
                              | (1) runtime error: invalid memory address or nil pointer dereference
                              | Error types: (1) runtime.errorString
                            Wraps: (3) panic applying step 
                              | db0.ScanForShare(ctx, tk(1854443122230575953), tk(17482680577027038983), 0): runtime error: invalid memory address or nil pointer dereference
                            Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
    panic.go:523: -- test log scope end --
test logs left over in: outputs.zip/logTestKVNemesisSingleNode_ReproposalChaos3538267343
--- FAIL: TestKVNemesisSingleNode_ReproposalChaos (4.13s)

Parameters: attempt=1 , run=27 , shard=1

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-33258

erikgrinaker commented 11 months ago

Dupe of #113924.

cockroach-teamcity commented 11 months ago

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on master @ 414cb46513079bb8869bd776f045ed6c96ad6d26:

github.com/cockroachdb/cockroach/pkg/rpc.clientStream.RecvMsg(...)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1394
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.(*tracingClientStream).RecvMsg(0xc0033e1f80, {0x4e69100?, 0xc00620d740?})
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:392 +0x31
github.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv({{0x6cd08d0?, 0xc0033e1f80?}})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1080 +0x53
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:863 +0x25
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc00524caf0, 0xc006925928)
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:135 +0x185
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc002136100, {0x6c9eb88?, 0xc007db2b40?}, 0xc003acbd40, {{0xc0041489f8, 0x1, 0x8}, {0xc006536450, 0x14, 0x18}}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:862 +0xb11
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc002136100, {0x6c9eb88, 0xc007db2b40}, 0xc003acbd40, {{0xc0041489f8, 0x1, 0x8}, {0xc006536450, 0x14, 0x18}}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:576 +0x69f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.func1.1({0x6c9eb88?, 0xc007db2b40?})
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:263 +0xf4
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.func1.Group.GoCtx.func2()
    github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x22
golang.org/x/sync/errgroup.(*Group).Go.func1()
    golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 13475
    golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0x96

goroutine 16830 [select]:
github.com/cockroachdb/cockroach/pkg/rpc.(*pipe).recv(0xc005d407a0, {0x6c9ebc0, 0xc006a4be60})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1310 +0xdc
github.com/cockroachdb/cockroach/pkg/rpc.pipeReader.recv({0xc005d407a0?, 0x5400f10?}, {0x6c9ebc0?, 0xc006a4be60?}, {0x4e69100, 0xc0071bcdb0})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1346 +0x3c
github.com/cockroachdb/cockroach/pkg/rpc.clientStream.RecvMsg(...)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1394
github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.(*tracingClientStream).RecvMsg(0xc002b3e880, {0x4e69100?, 0xc0071bcdb0?})
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:392 +0x31
github.com/cockroachdb/cockroach/pkg/rpc.rangeFeedClientAdapter.Recv({{0x6cd08d0?, 0xc002b3e880?}})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:1080 +0x53
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed.func3()
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:863 +0x25
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*stuckRangeFeedCanceler).do(0xc0049245a0, 0xc0026dd928)
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed_canceler.go:135 +0x185
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).singleRangeFeed(0xc002136100, {0x6c9eb88?, 0xc007db2b40?}, 0xc006290000, {{0xc008ab3a28, 0x14, 0x18}, {0xc004148a00, 0x1, 0x8}}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:862 +0xb11
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).partialRangeFeed(0xc002136100, {0x6c9eb88, 0xc007db2b40}, 0xc006290000, {{0xc008ab3a28, 0x14, 0x18}, {0xc004148a00, 0x1, 0x8}}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:576 +0x69f
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.func1.1({0x6c9eb88?, 0xc007db2b40?})
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender_rangefeed.go:263 +0xf4
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).RangeFeedSpans.func1.Group.GoCtx.func2()
    github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:168 +0x22
golang.org/x/sync/errgroup.(*Group).Go.func1()
    golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 13475
    golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:72 +0x96

Parameters: TAGS=bazel,gss , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 11 months ago

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on master @ 414cb46513079bb8869bd776f045ed6c96ad6d26:

                              |     github.com/cockroachdb/cockroach/pkg/server/node.go:1464
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704
                              | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12
                              |     github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
                              | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
                              | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
                              | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
                              | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
                              |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
                              | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
                            Wraps: (2) secondary error attachment
                              | runtime error: invalid memory address or nil pointer dereference
                              | (1) runtime error: invalid memory address or nil pointer dereference
                              | Error types: (1) runtime.errorString
                            Wraps: (3) panic applying step 
                              | db0.Txn(ctx, func(ctx context.Context, txn *kv.Txn) error {
                              |   txn.SetIsoLevel(isolation.ReadCommitted)
                              |   txn.Del(ctx, tk(4394514764636405117) /* @s132 */)
                              |   txn.ReverseScanForShareSkipLocked(ctx, tk(9552068926050814364), tk(14030239616544344760), 0) // @1699365857.839786059,0 <nil>
                              |   txn.ReverseScanForUpdate(ctx, tk(12794402163027398203), tk(15739822530194244867), 0) // @1699365857.853530876,0 <nil>
                              |   {
                              |     b := &kv.Batch{}
                              |     b.ReverseScanForShareGuaranteedDurability(tk(2093756760660622341), tk(14330641433808437563)) // TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "unnamed" meta={id=baebe378 key=/Table/100/"3cfc73e5325cbd7d" iso=ReadCommitted pri=0.01007281 epo=0 ts=1699365862.835739729,0 min=1699365857.830318692,0 seq=1} lock=true stat=ABORTED rts=1699365857.858082000,0 wto=false gul=1699365857.840318692,0
                              |     b.Del(tk(15542295355324478847) /* @s133 */) // TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "unnamed" meta={id=baebe378 key=/Table/100/"3cfc73e5325cbd7d" iso=ReadCommitted pri=0.01007281 epo=0 ts=1699365862.835739729,0 min=1699365857.830318692,0 seq=1} lock=true stat=ABORTED rts=1699365857.858082000,0 wto=false gul=1699365857.840318692,0
                              |     b.ScanForUpdate(tk(25853124770866442), tk(6985506453381389282)) // TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "unnamed" meta={id=baebe378 key=/Table/100/"3cfc73e5325cbd7d" iso=ReadCommitted pri=0.01007281 epo=0 ts=1699365862.835739729,0 min=1699365857.830318692,0 seq=1} lock=true stat=ABORTED rts=1699365857.858082000,0 wto=false gul=1699365857.840318692,0
                              |     txn.Run(ctx, b) // TransactionRetryWithProtoRefreshError: TransactionAbortedError(ABORT_REASON_PUSHER_ABORTED): "unnamed" meta={id=baebe378 key=/Table/100/"3cfc73e5325cbd7d" iso=ReadCommitted pri=0.01007281 epo=0 ts=1699365862.835739729,0 min=1699365857.830318692,0 seq=1} lock=true stat=ABORTED rts=1699365857.858082000,0 wto=false gul=1699365857.840318692,0
                              |   }
                              |   txn.ScanForShareSkipLockedGuaranteedDurability(ctx, tk(4584485228542261207), tk(12791311550360948978), 0) // omitted
                              |   b := &kv.Batch{}
                              |   b.GetForShare(tk(1444201270118969291)) // omitted
                              |   b.ScanForUpdateGuaranteedDurability(tk(4594628739714020331), tk(18076318797112915640)) // omitted
                              |   b.ReverseScanForShare(tk(6416714621173794618), tk(11826554832718319865)) // omitted
                              |   txn.CommitInBatch(ctx, b) // omitted
                              |   return nil
                              | }): runtime error: invalid memory address or nil pointer dereference
                            Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos3622758967
--- FAIL: TestKVNemesisSingleNode_ReproposalChaos (72.05s)

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

nvanbenschoten commented 11 months ago

Fixed by https://github.com/cockroachdb/cockroach/pull/113987.