search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.89k stars 3.77k forks source link

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #115598

Closed cockroach-teamcity closed 4 months ago

cockroach-teamcity commented 9 months ago

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ df4b4e856c33477cdad4a659adba959987c488ca:

Fatal error:

panic: lock table num=2
 lock: /Table/100/"be00911b49a6387e"
  holder: txn: b8955c33-8477-41d2-9aaf-32ef08feb78a epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 299, strength: Shared, txn: none
   distinguished req: 299
 lock: /Table/100/"f91ee672f2abf20d"
  holder: txn: 11525b9b-4822-4097-91a1-486697cd79e1 epoch: 0, iso: ReadCommitted, ts: 1701781195.664329888,1, info: repl [Intent]
   queued locking requests:
    active: true req: 254, strength: Intent, txn: none
   distinguished req: 254

error: queued locking request 299 does not conflict with holder/waiting requests  lock: /Table/100/"be00911b49a6387e"
  holder: txn: b8955c33-8477-41d2-9aaf-32ef08feb78a epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: true req: 299, strength: Shared, txn: none
   distinguished req: 299 [recovered]
    panic: lock table num=2
 lock: /Table/100/"be00911b49a6387e"
  holder: txn: b8955c33-8477-41d2-9aaf-32ef08feb78a epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 299, strength: Shared, txn: none
   distinguished req: 299
 lock: /Table/100/"f91ee672f2abf20d"
  holder: txn: 11525b9b-4822-4097-91a1-486697cd79e1 epoch: 0, iso: ReadCommitted, ts: 1701781195.664329888,1, info: repl [Intent]
   queued locking requests:
    active: true req: 254, strength: Intent, txn: none
   distinguished req: 254

error: queued locking request 299 does not conflict with holder/waiting requests  lock: /Table/100/"be00911b49a6387e"
  holder: txn: b8955c33-8477-41d2-9aaf-32ef08feb78a epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: true req: 299, strength: Shared, txn: none
   distinguished req: 299

Stack: 

goroutine 3112257 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc0122c76e0?, {0x6d9be30, 0xc013d5ec00})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x48efba0, 0xc00f27c2e0})
    GOROOT/src/runtime/panic.go:884 +0x213
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc0003f0280)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x20d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.UpdateLocks({{0x7fda51a7bca0?, 0xc0003f0280?}}, 0x1?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:99 +0x94
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).OnLockUpdated(0x52439d8?, {0x6d9be30, 0xc00fa75cb0}, 0x6?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:552 +0x36
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc002806580, {0x6d9be30, 0xc00fa75cb0}, {0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:793 +0x802
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects(0xc002806688, {0x6d9be30, 0xc00fa75cb0}, {0x6ddaac8?, 0xc004054008})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:222 +0x848
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter({0x7fda4f599c20?, 0xc002806a68}, 0xc0122c8570)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214 +0x158
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc0122c8ae8, {0x6d9be30, 0xc0103c85a0}, {0x6dcae20, 0xc002806a08})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295 +0x1f8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc0122c8ae8, {0x6d9be30, 0xc0103c85a0})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x9a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1053 +0x1425
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:740 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc006824000, 0xc0170d71a0?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc000fd9f10, {0x6d9be30, 0xc013d5ec00}, {0x6d81820, 0xc006824000}, 0xc00f261500)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x195
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6d9be30?, 0xc013d5ec00?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
Log preceding fatal error

``` === RUN TestKVNemesisSingleNode_ReproposalChaos test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos734143989 test_log_scope.go:81: use -show-logs to present logs inline kvnemesis_test.go:283: seed: 750006408177795411 kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis429571242 ```

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-34118

erikgrinaker commented 9 months ago

@arulajmani Related to the recent lock table assertions you've added?

cockroach-teamcity commented 9 months ago

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ df4b4e856c33477cdad4a659adba959987c488ca:

                              | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
                              | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
                              | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
                              |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
                              | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
                              | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714
                              | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2
                              |     github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
                              | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899
                            Wraps: (2) panic applying step 
                              | db0.Txn(ctx, func(ctx context.Context, txn *kv.Txn) error {
                              |   txn.SetIsoLevel(isolation.ReadCommitted)
                              |   txn.DelRange(ctx, tk(8543534448051736489), tk(9408248879097924965), true /* @s114 */) // @1701782463.701752092,0 <nil>
                              |   txn.ReverseScanForUpdateGuaranteedDurability(ctx, tk(10993648730429643591), tk(18161759648309122977), 0) // @1701782463.703782197,0 (/Table/100/"f9257238331e0830":v34, /Table/100/"ed3e32e24bc0dc5c":v34, <nil>)
                              |   txn.ReverseScanForShareGuaranteedDurability(ctx, tk(8543631286166663424), tk(17421013793695485217), 0)
                              |   txn.Get(ctx, tk(17762943607935793602))
                              |   return errors.New("rollback")
                              | }): lock table num=4
                              |  lock: /Table/100/"3c2b77024c8063fd"
                              |   holder: txn: b59e7e22-7741-4fbe-873c-059a6696c11c epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
                              |    queued locking requests:
                              |     active: true req: 457, strength: Shared, txn: none
                              |    distinguished req: 457
                              |  lock: /Table/100/"47da129319a8446b"
                              |   holder: txn: b59e7e22-7741-4fbe-873c-059a6696c11c epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
                              |  lock: /Table/100/"5967f24f663056f2"
                              |   holder: txn: b59e7e22-7741-4fbe-873c-059a6696c11c epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
                              |  lock: /Table/100/"5d031b446cfd8fa1"
                              |   holders: txn: b59e7e22-7741-4fbe-873c-059a6696c11c epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
                              |            txn: 1a6e8dba-16a0-42ca-9196-9f94736f2cb2 epoch: 0, iso: ReadCommitted, info: repl [Shared]
                              |    queued locking requests:
                              |     active: true req: 453, strength: Intent, txn: none
                              |    distinguished req: 453
                              |
                              | error: queued locking request 457 does not conflict with holder/waiting requests  lock: /Table/100/"3c2b77024c8063fd"
                              |   holder: txn: b59e7e22-7741-4fbe-873c-059a6696c11c epoch: 0, iso: ReadCommitted, info: repl [Shared]
                              |    queued locking requests:
                              |     active: true req: 457, strength: Shared, txn: none
                              |    distinguished req: 457
                            Error types: (1) *withstack.withStack (2) *errutil.leafError
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos1839409175
--- FAIL: TestKVNemesisSingleNode_ReproposalChaos (44.22s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

arulajmani commented 9 months ago

Yeah, this looks related to the verification I added. Looking into this, but I'll re-label this as a GA-blocker given we're only seeing this with shared locks.

arulajmani commented 9 months ago

I have a plausible explanation for what might be going on here. In both these cases, we see the panic is coming from a lock held by a finalized transaction. For example, 

                              |  lock: /Table/100/"3c2b77024c8063fd"
                              |   holder: txn: b59e7e22-7741-4fbe-873c-059a6696c11c epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
                              |    queued locking requests:
                              |     active: true req: 457, strength: Shared, txn: none
                              |    distinguished req: 457

Note that the shared lock held by txn b59e7e22 is replicated. That means it must have been pulled into the lock table by a conflicting request. My guess is that this must be a non-transactional[1] exclusive/intent locking request.

That would explain why req 457 is waiting on this key. Once txn b59e7e22 is finalized, the non-transactional exclusive/intent locking request returns from its push and resumes its scan. As it no longer conflicts with the lock, and its a non-transactional request, its removed from the lock's wait queue. This would then explain the lock state we see the test fail with.

This isn't really a correctness issue, even though we're tripping verification here. To that end, I'll remove the GA-blocker label. I'm not sure what the best fix should be here yet.

[1] If it were a transactional request, we'd expect it to still be in the lock's wait queue as an inactive waiter.

cockroach-teamcity commented 9 months ago

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ 75dbd5b7dd05e885fd729e3bb13622c8f8b74b85:

Fatal error:

panic: lock table num=6
 lock: /Table/100/"cc4dd6d011c63d0b"
   queued locking requests:
    active: false req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
 lock: /Table/100/"cd1ed982d875f840"
   queued locking requests:
    active: false req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
 lock: /Table/100/"d836d47b42b9fc8c"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
   distinguished req: 62
 lock: /Table/100/"e55f9f6ba0074cee"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e7d992cf5d406ad2"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e8486f53f08cba7a"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]

error: queued locking request 62 does not conflict with holder/waiting requests  lock: /Table/100/"d836d47b42b9fc8c"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: true req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
   distinguished req: 62 [recovered]
    panic: lock table num=6
 lock: /Table/100/"cc4dd6d011c63d0b"
   queued locking requests:
    active: false req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
 lock: /Table/100/"cd1ed982d875f840"
   queued locking requests:
    active: false req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
 lock: /Table/100/"d836d47b42b9fc8c"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
   distinguished req: 62
 lock: /Table/100/"e55f9f6ba0074cee"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e7d992cf5d406ad2"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e8486f53f08cba7a"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared] [holder finalized: aborted]

error: queued locking request 62 does not conflict with holder/waiting requests  lock: /Table/100/"d836d47b42b9fc8c"
  holder: txn: 729bac53-5106-44e9-822b-eba41f1440cc epoch: 0, iso: Serializable, info: repl [Shared]
   queued locking requests:
    active: true req: 62, strength: Shared, txn: af6a04ea-75d1-4f85-a41f-8e109ba383a8
   distinguished req: 62

Stack: 

goroutine 3177922 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x200b86a?, {0x6d9def0, 0xc00862bc20})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x48f14a0, 0xc00bb90d60})
    GOROOT/src/runtime/panic.go:884 +0x213
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc000800f00)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x20d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.UpdateLocks({{0x7f1e824b2450?, 0xc000800f00?}}, 0x1?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:99 +0x94
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).OnLockUpdated(0x52455f1?, {0x6d9def0, 0xc001144960}, 0x6?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:552 +0x36
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleReadWriteLocalEvalResult(0xc0098c2c80, {0x6d9def0, 0xc001144960}, {0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:793 +0x802
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects(0xc0098c2d88, {0x6d9def0, 0xc001144960}, {0x6ddcb88?, 0xc002924508})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:222 +0x848
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter({0x7f1e83252740?, 0xc0098c3168}, 0xc003fec570)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:214 +0x158
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc003fecae8, {0x6d9def0, 0xc015f66990}, {0x6dccee0, 0xc0098c3108})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295 +0x1f8
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc003fecae8, {0x6d9def0, 0xc015f66990})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x9a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1053 +0x1425
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:740 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc0059f0a80, 0xc012256ae0?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc00d6b75e0, {0x6d9def0, 0xc00862bc20}, {0x6d838e0, 0xc0059f0a80}, 0xc00825b500)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x195
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6d9def0?, 0xc00862bc20?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
Log preceding fatal error

``` === RUN TestKVNemesisSingleNode_ReproposalChaos test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos125762160 test_log_scope.go:81: use -show-logs to present logs inline kvnemesis_test.go:283: seed: 29713711782940073 kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1233562520 ```

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ 43a2532550ca7abfc3c5682df31b0cc20da934b0:

                              | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
                              | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
                              |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
                              | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
                              | github.com/cockroachdb/cockroach/pkg/rpc.bindUnaryServerInterceptorToHandler.func1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714
                              | github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor.ClientInterceptor.func2
                              |     github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
                              | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899
                            Wraps: (2) panic applying step 
                              | db0.Txn(ctx, func(ctx context.Context, txn *kv.Txn) error {
                              |   txn.SetIsoLevel(isolation.Snapshot)
                              |   txn.ScanForShare(ctx, tk(10161750788992158904), tk(10607603416092531769), 0) // @1702292838.076731702,0 <nil>
                              |   txn.ReverseScanForShareSkipLocked(ctx, tk(12325901490205422963), tk(16583471604304201683), 0) // @1702292838.076731702,0 (/Table/100/"e4a079ee176e85ad":v85, /Table/100/"e31eb55758d3c35a":v89, <nil>)
                              |   txn.Get(ctx, tk(3403608216299324618)) // @1702292838.076731702,0 (<nil>, <nil>)
                              |   txn.GetForUpdate(ctx, tk(17532841644207285003)) // @1702292838.076731702,0 (<nil>, <nil>)
                              |   txn.ReverseScanForUpdate(ctx, tk(6132161645053430098), tk(7977877486819318195), 0) // @1702292838.076731702,0 (/Table/100/"6a00e953313860af":v72, <nil>)
                              |   b := &kv.Batch{}
                              |   b.ReverseScanForShareGuaranteedDurability(tk(6560877028758136323), tk(12124959853369502821))
                              |   b.ReverseScanForUpdateGuaranteedDurability(tk(6822100593481620689), tk(8187737862183061801))
                              |   txn.CommitInBatch(ctx, b)
                              |   return nil
                              | }): lock table num=4
                              |  lock: /Table/100/"41a761029fa3b064"
                              |   holder: txn: 0f329068-7844-47d2-81f8-03606ad9b79c epoch: 0, iso: Snapshot, info: repl [Shared]
                              |  lock: /Table/100/"4963dbe46f3295b0"
                              |   holder: txn: 0f329068-7844-47d2-81f8-03606ad9b79c epoch: 0, iso: Snapshot, info: repl [Shared]
                              |  lock: /Table/100/"6a00e953313860af"
                              |   holder: txn: 0f329068-7844-47d2-81f8-03606ad9b79c epoch: 0, iso: Snapshot, info: repl [Shared]
                              |    queued locking requests:
                              |     active: true req: 152, strength: Shared, txn: e9314727-f793-4592-a65e-fbf6804c6837
                              |    distinguished req: 152
                              |  lock: /Table/100/"7161200d7eff3df8"
                              |    queued locking requests:
                              |     active: false req: 152, strength: Exclusive, txn: e9314727-f793-4592-a65e-fbf6804c6837
                              |
                              | error: queued locking request 152 does not conflict with holder/waiting requests  lock: /Table/100/"6a00e953313860af"
                              |   holder: txn: 0f329068-7844-47d2-81f8-03606ad9b79c epoch: 0, iso: Snapshot, info: repl [Shared]
                              |    queued locking requests:
                              |     active: true req: 152, strength: Shared, txn: e9314727-f793-4592-a65e-fbf6804c6837
                              |    distinguished req: 152
                            Error types: (1) *withstack.withStack (2) *errutil.leafError
    panic.go:522: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos2936116515
--- FAIL: TestKVNemesisSingleNode_ReproposalChaos (45.92s)

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ caaa842c957d8487be32504cdd6379745f1a3ff6:

Fatal error:

panic: lock table num=20
 lock: /Table/100/"b2d30b8a0853de46"
  holder: txn: 4c91b6e2-b540-4418-a524-fb0d98631feb epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: false req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
    active: true req: 25, strength: Exclusive, txn: 0f10bc94-1797-4806-ae47-2b8fbc358cf3
   distinguished req: 25
 lock: /Table/100/"bef75b44f626448c"
  holder: txn: 4c91b6e2-b540-4418-a524-fb0d98631feb epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: false req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
 lock: /Table/100/"c9682ffc41c21c21"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
   distinguished req: 21
 lock: /Table/100/"e0c6ae0ae7889ef0"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e0db12f1599718f8"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e14682aeacfaa0da"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e1a38d86c350b2cf"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e1bcdaf993d23b26"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e2e1ff711fa566ed"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e3bc17591f7941fe"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e4a630f18f44509b"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e62ce91b7825c690"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e6622f653b6dfce0"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e7b8227548d1a1b1"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e86983ea7356c4d5"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e910e5cf4cd49bd3"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"eab81f7152d0ec32"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"ee6e807e54c65f1f"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"ee855cac35ddba9e"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"eea34826ecb3ddc0"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]

error: queued locking request 21 does not conflict with holder/waiting requests  lock: /Table/100/"c9682ffc41c21c21"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
   distinguished req: 21 [recovered]
    panic: lock table num=20
 lock: /Table/100/"b2d30b8a0853de46"
  holder: txn: 4c91b6e2-b540-4418-a524-fb0d98631feb epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: false req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
    active: true req: 25, strength: Exclusive, txn: 0f10bc94-1797-4806-ae47-2b8fbc358cf3
   distinguished req: 25
 lock: /Table/100/"bef75b44f626448c"
  holder: txn: 4c91b6e2-b540-4418-a524-fb0d98631feb epoch: 0, iso: ReadCommitted, info: repl [Shared]
   queued locking requests:
    active: false req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
 lock: /Table/100/"c9682ffc41c21c21"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
   distinguished req: 21
 lock: /Table/100/"e0c6ae0ae7889ef0"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e0db12f1599718f8"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e14682aeacfaa0da"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e1a38d86c350b2cf"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e1bcdaf993d23b26"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e2e1ff711fa566ed"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e3bc17591f7941fe"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e4a630f18f44509b"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e62ce91b7825c690"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e6622f653b6dfce0"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e7b8227548d1a1b1"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e86983ea7356c4d5"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"e910e5cf4cd49bd3"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"eab81f7152d0ec32"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"ee6e807e54c65f1f"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"ee855cac35ddba9e"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"eea34826ecb3ddc0"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]

error: queued locking request 21 does not conflict with holder/waiting requests  lock: /Table/100/"c9682ffc41c21c21"
  holder: txn: 8fdf97e6-2449-452e-a3e9-b2396f978847 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 21, strength: Shared, txn: 84752f9b-90e7-4605-858f-26188a64a348
   distinguished req: 21

Stack: 

goroutine 2201668 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x200ea2a?, {0x6da62f0, 0xc00ac74960})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x6a
panic({0x48f79c0, 0xc00c903970})
    GOROOT/src/runtime/panic.go:884 +0x213
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc0105a2dc0)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x20d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.Dequeue({{0x7f16183d2088?, 0xc0105a2dc0?}}, {0x6dd5418?, 0xc004112280?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:77 +0x91
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).FinishReq(0xc005a00640?, 0xc0009f5bc0)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:453 +0x106
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*endCmds).done(0xc00b4bb960, {0x6da62f0?, 0xc008305350?}, 0x0?, 0x0?, 0x0?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:1367 +0x143
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*ProposalData).finishApplication(0xc00b4bb900, {0x6da62f0?, 0xc008305350?}, {0xc01436a880, 0x0, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:242 +0x53
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicatedCmd).AckOutcomeAndFinish(0xc001ed6508, {0x6da62f0?, 0xc008305350?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_cmd.go:156 +0xa5
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.forEachAppliedCmdIter({0x6da62f0, 0xc008305350}, {0x7f1617b6edd0?, 0xc0093096b8?}, 0x55033b8)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:268 +0xea
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc00076cae8, {0x6da62f0, 0xc008305350}, {0x6dd5340, 0xc009309688})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:301 +0x255
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc00076cae8, {0x6da62f0, 0xc008305350})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x9a
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1053 +0x1425
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:740 +0x1b0
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc0041cb500, 0xc0110fc090?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x14b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc0010eb3b0, {0x6da62f0, 0xc00ac74960}, {0x6d8bce0, 0xc0041cb500}, 0xc00fcfc000)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x195
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6da62f0?, 0xc00ac74960?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x45
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x146
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x43b
Log preceding fatal error

``` === RUN TestKVNemesisSingleNode_ReproposalChaos test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos1525528943 test_log_scope.go:81: use -show-logs to present logs inline kvnemesis_test.go:283: seed: 6622495634950520862 kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3805446582 ```

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

cockroach-teamcity commented 9 months ago

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ bc665a398d2ddbd27b60119e8272f49d0abf78a2:

                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
                              | github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
                              | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136
                              | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
                              |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336
                              | github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714
                              | github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8
                              |     github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228
                              | github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899
                              | github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784
                              | github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch
                              |     github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907
                            Wraps: (2) panic applying step 
                              | db0.ReverseScanForUpdate(ctx, tk(5234225380589050268), tk(7787489959389179047), 0): lock table num=4
                              |  lock: /Table/100/"166554c6bbe0c7f2"
                              |   holder: txn: decf2ce2-ea91-47fa-8813-298360ed21ab epoch: 0, iso: ReadCommitted, info: repl [Shared] [holder finalized: aborted]
                              |    queued locking requests:
                              |     active: true req: 370, strength: Shared, txn: none
                              |    distinguished req: 370
                              |  lock: /Table/100/"2b18e1a7ddcdec02"
                              |   holder: txn: b242c9ba-a917-493d-a1e4-9bd2c5711ce7 epoch: 0, iso: Serializable, ts: 1702478437.087277341,0, info: repl [Intent]
                              |    queued locking requests:
                              |     active: true req: 361, strength: Intent, txn: none
                              |    distinguished req: 361
                              |  lock: /Table/100/"9f4bff7f0c26631c"
                              |   holder: txn: 6c4f93af-3ac0-4629-8a8b-ca20488b6523 epoch: 0, iso: ReadCommitted, info: unrepl [(str: Shared seq: 0)]
                              |  lock: /Table/100/"c1ff33290a82bc24"
                              |   holder: txn: 6c4f93af-3ac0-4629-8a8b-ca20488b6523 epoch: 0, iso: ReadCommitted, info: unrepl [(str: Shared seq: 0)]
                              |
                              | error: queued locking request 370 does not conflict with holder/waiting requests  lock: /Table/100/"166554c6bbe0c7f2"
                              |   holder: txn: decf2ce2-ea91-47fa-8813-298360ed21ab epoch: 0, iso: ReadCommitted, info: repl [Shared]
                              |    queued locking requests:
                              |     active: true req: 370, strength: Shared, txn: none
                              |    distinguished req: 370
                            Error types: (1) *withstack.withStack (2) *errutil.leafError
    panic.go:523: -- test log scope end --
test logs left over in: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos4110185987
--- FAIL: TestKVNemesisSingleNode_ReproposalChaos (8.70s)

Parameters: TAGS=bazel,gss , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #116322 kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed [C-test-failure O-robot T-kv branch-release-23.2.0-rc release-blocker]

This test on roachdash | Improve this report!

github-actions[bot] commented 6 months ago

We have marked this test failure issue as stale because it has been inactive for 1 month. If this failure is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 5 days to keep the test failure queue tidy.

cockroach-teamcity commented 6 months ago

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2 @ 84f0e5f4625f8a2155d531251e0df6db8a5ef053:

Fatal error:

panic: r80/1: closed timestamp regression 1709903797.620507717,0 -> 1709903797.618308551,0 (cmd "2d34d0a7edc23f62" index 15 term 6 (LAI=2 CTS=1709903797.618308551,0))
data: 002d34d0a7edc23f6230016a154200920110098cb1ce7672ccba17800102a8018602729e010a9b01000000000000000001000000012e017a6b12ec123565636338306539346531376635316600ff01000100019f7a0c7469324976b4e9d767de021fad125e0a460a109f7a0c7469324976b4e9d767de021fad1a14ec123165313132323636646233346335336400012a0a088ce3bab6a78eb3dd1730e1ce324a0a088ce3bab6a78eb3dd175001120c088ce3bab6a78eb3dd17100018002000280048017a008201260a12637264622d7472616365722d7370616e69641210313236613666343533313638363939638201270a13637264622d7472616365722d747261636569641210343837626665646230643136306536378201080a0372656312017620028a010a08c7eb85c99c8eb3dd17 [recovered]
    panic: r80/1: closed timestamp regression 1709903797.620507717,0 -> 1709903797.618308551,0 (cmd "2d34d0a7edc23f62" index 15 term 6 (LAI=2 CTS=1709903797.618308551,0))
data: 002d34d0a7edc23f6230016a154200920110098cb1ce7672ccba17800102a8018602729e010a9b01000000000000000001000000012e017a6b12ec123565636338306539346531376635316600ff01000100019f7a0c7469324976b4e9d767de021fad125e0a460a109f7a0c7469324976b4e9d767de021fad1a14ec123165313132323636646233346335336400012a0a088ce3bab6a78eb3dd1730e1ce324a0a088ce3bab6a78eb3dd175001120c088ce3bab6a78eb3dd17100018002000280048017a008201260a12637264622d7472616365722d7370616e69641210313236613666343533313638363939638201270a13637264622d7472616365722d747261636569641210343837626665646230643136306536378201080a0372656312017620028a010a08c7eb85c99c8eb3dd17

Stack: 

goroutine 11038 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1f14b45?, {0x6c21ae0, 0xc005e65050})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x46d79c0?, 0xc004e30350?})
    GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*rangeAsserter).apply.func1({0x5065b6d?, 0xc005ea4ae0?}, {0xc00442f9f0?, 0x8?, 0xc00442f650?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/asserter.go:250 +0x165
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*rangeAsserter).apply(0xc006c98c80, 0x1, {0xc0063647d8, 0x8}, {0x6, 0xf, 0x0, {0xc00973e160, 0x132, 0x143}}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/asserter.go:301 +0x5ff
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Asserter).Apply(0x4fde1c?, 0x89122448912244?, 0x492001?, {0xc0063647d8, 0x8}, {0x6, 0xf, 0x0, {0xc00973e160, 0x132, ...}}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/asserter.go:231 +0xa5
github.com/cockroachdb/cockroach/pkg/kv/kvnemesis.kvnemesisTestCfg.testClusterArgs.func7({{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, {0x0, ...}, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvnemesis/pkg/kv/kvnemesis/kvnemesis_test.go:145 +0x86
github.com/cockroachdb/cockroach/pkg/kv/kvserver.replicaApplyTestingFilters({0xc001e2ad90?, 0x6c21ae0?}, 0xc004cd3bf0?, 0xc001a78248?, {0xc001e2ade0?, 0xc005754640?, 0x0?}, 0xe0?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:117 +0x1f7
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaAppBatch).Stage(0xc001e2ad90, {0x6c21ae0, 0xc004cd3bf0}, {0x6c41a80?, 0xc001a78248})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_app_batch.go:113 +0xe9
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCmdIter({0x6c50cb0?, 0xc001e2b138}, 0xc004430598)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:192 +0x148
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc004430af8, {0x6c21ae0, 0xc005d47f20}, {0x6c50cb0, 0xc001e2b108})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:284 +0x150
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc004430af8, {0x6c21ae0, 0xc005d47f20})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x95
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1059 +0x13c5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:744 +0x1b8
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc0029f8a80, 0xc0052aeae0?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x12b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc003668540, {0x6c21ae0, 0xc005e65050}, {0x6c05c60, 0xc0029f8a80}, 0xc0029f8000)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x1e6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6c21ae0?, 0xc005e65050?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 10752
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415
Log preceding fatal error

``` === RUN TestKVNemesisSingleNode_ReproposalChaos test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos2182480276 test_log_scope.go:81: use -show-logs to present logs inline kvnemesis_test.go:319: seed: 6437887958677442706 kvnemesis_test.go:226: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis2296613533 ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

github-actions[bot] commented 5 months ago

We have marked this test failure issue as stale because it has been inactive for 1 month. If this failure is still relevant, removing the stale label or adding a comment will keep it active. Otherwise, we'll close it in 5 days to keep the test failure queue tidy.