search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.09k stars 3.81k forks source link

kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed #116322

Closed cockroach-teamcity closed 10 months ago

cockroach-teamcity commented 10 months ago

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2.0-rc @ 17da6cf959356fe67d5650c9d404ef2e772bee10:

Fatal error:

panic: lock table num=10
 lock: /Table/100/"29c924e1b7b6dded"
   queued locking requests:
    active: false req: 257, strength: Shared, txn: 4ca9f714-9a11-45fa-9df3-3bb713726ce6
 lock: /Table/100/"44f20ae691ab2be1"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 257, strength: Shared, txn: 4ca9f714-9a11-45fa-9df3-3bb713726ce6
   distinguished req: 257
 lock: /Table/100/"46976a6180dfaf52"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"4717602b6dbb83a3"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"5a9e33ff9a55eae0"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"6a5094f10190360a"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"7cf942e5457697a2"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"84ff904534c39af4"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"bf14ba4774da76cd"
  holder: txn: 383c49c1-2b5d-4a48-93fc-3b3ec09f6fee epoch: 0, iso: Snapshot, info: unrepl [(str: Shared seq: 0)]
   queued locking requests:
    active: true req: 216, strength: Intent, txn: none
   distinguished req: 216
 lock: /Table/100/"ca3c6ebb036db8da"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: false req: 216, strength: Intent, txn: none
    active: true req: 256, strength: Exclusive, txn: none
   distinguished req: 256

error: queued locking request 257 does not conflict with holder/waiting requests  lock: /Table/100/"44f20ae691ab2be1"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 257, strength: Shared, txn: 4ca9f714-9a11-45fa-9df3-3bb713726ce6
   distinguished req: 257 [recovered]
    panic: lock table num=10
 lock: /Table/100/"29c924e1b7b6dded"
   queued locking requests:
    active: false req: 257, strength: Shared, txn: 4ca9f714-9a11-45fa-9df3-3bb713726ce6
 lock: /Table/100/"44f20ae691ab2be1"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: true req: 257, strength: Shared, txn: 4ca9f714-9a11-45fa-9df3-3bb713726ce6
   distinguished req: 257
 lock: /Table/100/"46976a6180dfaf52"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"4717602b6dbb83a3"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"5a9e33ff9a55eae0"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"6a5094f10190360a"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"7cf942e5457697a2"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"84ff904534c39af4"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
 lock: /Table/100/"bf14ba4774da76cd"
  holder: txn: 383c49c1-2b5d-4a48-93fc-3b3ec09f6fee epoch: 0, iso: Snapshot, info: unrepl [(str: Shared seq: 0)]
   queued locking requests:
    active: true req: 216, strength: Intent, txn: none
   distinguished req: 216
 lock: /Table/100/"ca3c6ebb036db8da"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared] [holder finalized: aborted]
   queued locking requests:
    active: false req: 216, strength: Intent, txn: none
    active: true req: 256, strength: Exclusive, txn: none
   distinguished req: 256

error: queued locking request 257 does not conflict with holder/waiting requests  lock: /Table/100/"44f20ae691ab2be1"
  holder: txn: 4f704190-56e9-4566-bf86-352d96aaaf84 epoch: 0, iso: Snapshot, info: repl [Shared]
   queued locking requests:
    active: true req: 257, strength: Shared, txn: 4ca9f714-9a11-45fa-9df3-3bb713726ce6
   distinguished req: 257

Stack: 

goroutine 9844 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x1ed54c5?, {0x6b4eae0, 0xc0086c06f0})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x468e860?, 0xc00a5df670?})
    GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc00155fb80)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x1d4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.Dequeue({{0x7fb92408d9d0?, 0xc00155fb80?}}, {0x6b7d1d8?, 0xc005097540?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:77 +0x86
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).FinishReq(0xc00986b0e0?, 0xc0022cecc0)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:453 +0x102
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*endCmds).done(0xc005d8c060, {0x6b4eae0?, 0xc001149680?}, 0x0?, 0x0?, 0x0?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:1367 +0x13c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*ProposalData).finishApplication(0xc005d8c000, {0x6b4eae0?, 0xc001149680?}, {0x0, 0xc0025ee480, {0x0, 0x0, 0x0}, {0x0, 0x0, ...}})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal.go:242 +0x4d
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicatedCmd).AckOutcomeAndFinish(0xc00437e008, {0x6b4eae0?, 0xc001149680?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_cmd.go:156 +0x97
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.forEachAppliedCmdIter({0x6b4eae0, 0xc001149680}, {0x7fb92402e0d8?, 0xc0024df6b8}, 0x52cbfb8)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:268 +0xdd
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch(0xc0056d0b08, {0x6b4eae0, 0xc001149680}, {0x6b7da70, 0xc0024df688})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:301 +0x23e
github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries(0xc0056d0b08, {0x6b4eae0, 0xc001149680})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251 +0x95
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1053 +0x13c5
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:740 +0x1b8
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0xc004d04a80, 0xc0011d58c0?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689 +0x12b
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0xc007772230, {0x6b4eae0, 0xc0086c06f0}, {0x6b32d00, 0xc004d04a80}, 0xc004d04000)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418 +0x1e6
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x6b4eae0?, 0xc0086c06f0?})
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321 +0x46
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 9634
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415
Log preceding fatal error

``` === RUN TestKVNemesisSingleNode_ReproposalChaos test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos3380842340 test_log_scope.go:81: use -show-logs to present logs inline kvnemesis_test.go:283: seed: 5906600774162626727 kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis1991165452 ```

Parameters: TAGS=bazel,gss , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #115598 kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed [A-kv-transactions C-bug C-test-failure O-robot T-kv branch-release-23.2]

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-34584

cockroach-teamcity commented 10 months ago

kv/kvnemesis.TestKVNemesisSingleNode_ReproposalChaos failed with artifacts on release-23.2.0-rc @ 17da6cf959356fe67d5650c9d404ef2e772bee10:

Fatal error:

panic: lock table num=2
 lock: /Table/100/"adaf1171d1eab751"
  holder: txn: 66a8cfba-0029-4202-82cd-bff40773b2e9 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 1)]
   queued locking requests:
    active: true req: 83, strength: Shared, txn: 3c35e6c9-8ae7-468a-ba4b-da28026ee97b
    active: true req: 84, strength: Exclusive, txn: 54cdf8d3-4160-42de-9d51-01b8f43d7ab7
   distinguished req: 84
 lock: /Table/100/"bfe791e22a51cc52"
   queued locking requests:
    active: false req: 83, strength: Exclusive, txn: 3c35e6c9-8ae7-468a-ba4b-da28026ee97b

error: queued locking request 83 does not conflict with holder/waiting requests  lock: /Table/100/"adaf1171d1eab751"
  holder: txn: 66a8cfba-0029-4202-82cd-bff40773b2e9 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 1)]
   queued locking requests:
    active: true req: 83, strength: Shared, txn: 3c35e6c9-8ae7-468a-ba4b-da28026ee97b
    active: true req: 84, strength: Exclusive, txn: 54cdf8d3-4160-42de-9d51-01b8f43d7ab7
   distinguished req: 84 [recovered]
    panic: lock table num=2
 lock: /Table/100/"adaf1171d1eab751"
  holder: txn: 66a8cfba-0029-4202-82cd-bff40773b2e9 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 1)]
   queued locking requests:
    active: true req: 83, strength: Shared, txn: 3c35e6c9-8ae7-468a-ba4b-da28026ee97b
    active: true req: 84, strength: Exclusive, txn: 54cdf8d3-4160-42de-9d51-01b8f43d7ab7
   distinguished req: 84
 lock: /Table/100/"bfe791e22a51cc52"
   queued locking requests:
    active: false req: 83, strength: Exclusive, txn: 3c35e6c9-8ae7-468a-ba4b-da28026ee97b

error: queued locking request 83 does not conflict with holder/waiting requests  lock: /Table/100/"adaf1171d1eab751"
  holder: txn: 66a8cfba-0029-4202-82cd-bff40773b2e9 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 1)]
   queued locking requests:
    active: true req: 83, strength: Shared, txn: 3c35e6c9-8ae7-468a-ba4b-da28026ee97b
    active: true req: 84, strength: Exclusive, txn: 54cdf8d3-4160-42de-9d51-01b8f43d7ab7
   distinguished req: 84 [recovered]
    panic: lock table num=2
 lock: /Table/100/"adaf1171d1eab751"
  holder: txn: 66a8cfba-0029-4202-82cd-bff40773b2e9 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 1)]
   queued locking requests:
    active: true req: 83, strength: Shared, txn: 3c35e6c9-8ae7-468a-ba4b-da28026ee97b
    active: true req: 84, strength: Exclusive, txn: 54cdf8d3-4160-42de-9d51-01b8f43d7ab7
   distinguished req: 84
 lock: /Table/100/"bfe791e22a51cc52"
   queued locking requests:
    active: false req: 83, strength: Exclusive, txn: 3c35e6c9-8ae7-468a-ba4b-da28026ee97b

error: queued locking request 83 does not conflict with holder/waiting requests  lock: /Table/100/"adaf1171d1eab751"
  holder: txn: 66a8cfba-0029-4202-82cd-bff40773b2e9 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 1)]
   queued locking requests:
    active: true req: 83, strength: Shared, txn: 3c35e6c9-8ae7-468a-ba4b-da28026ee97b
    active: true req: 84, strength: Exclusive, txn: 54cdf8d3-4160-42de-9d51-01b8f43d7ab7
   distinguished req: 84 [recovered]
    panic: lock table num=2
 lock: /Table/100/"adaf1171d1eab751"
  holder: txn: 66a8cfba-0029-4202-82cd-bff40773b2e9 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 1)]
   queued locking requests:
    active: true req: 83, strength: Shared, txn: 3c35e6c9-8ae7-468a-ba4b-da28026ee97b
    active: true req: 84, strength: Exclusive, txn: 54cdf8d3-4160-42de-9d51-01b8f43d7ab7
   distinguished req: 84
 lock: /Table/100/"bfe791e22a51cc52"
   queued locking requests:
    active: false req: 83, strength: Exclusive, txn: 3c35e6c9-8ae7-468a-ba4b-da28026ee97b

error: queued locking request 83 does not conflict with holder/waiting requests  lock: /Table/100/"adaf1171d1eab751"
  holder: txn: 66a8cfba-0029-4202-82cd-bff40773b2e9 epoch: 0, iso: Serializable, info: unrepl [(str: Shared seq: 1)]
   queued locking requests:
    active: true req: 83, strength: Shared, txn: 3c35e6c9-8ae7-468a-ba4b-da28026ee97b
    active: true req: 84, strength: Exclusive, txn: 54cdf8d3-4160-42de-9d51-01b8f43d7ab7
   distinguished req: 84

Stack: 

goroutine 4785905 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc016261968?, {0x6b55ce0, 0xc00ccf4840})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x46971c0?, 0xc00d8a2220?})
    GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0xc002502000?, {0x6b55ce0, 0xc00c9743c0})
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:230 +0x65
panic({0x46971c0?, 0xc00d8a2220?})
    GOROOT/src/runtime/panic.go:914 +0x21f
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes.func1()
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:109 +0x14e
panic({0x46971c0?, 0xc00d8a2220?})
    GOROOT/src/runtime/panic.go:920 +0x270
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*lockTableImpl).verify(0xc0076a5900)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/lock_table.go:4710 +0x20d
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.verifyingLockTable.ScanAndEnqueue({{0x7fa1bcfee5f0?, 0xc0076a5900?}}, {0xc010644ea0, {0x17a06dc83160d618, 0x0, 0x0}, 0x0, 0x0, 0x0, 0x0, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/verifiable_lock_table.go:64 +0x131
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).sequenceReqWithGuard(0xc007252c30, {0x6b55ce0, 0xc00c9ac720}, 0xc00219a780, 0xc00625f380?)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:337 +0x856
github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency.(*managerImpl).SequenceReq(0x1?, {0x6b55ce0, 0xc00c9ac720}, 0xe?, {0xc010644ea0, {0x17a06dc83160d618, 0x0, 0x0}, 0x0, 0x0, ...}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/concurrency/concurrency_manager.go:241 +0x2cc
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).executeBatchWithConcurrencyRetries(0xc00ae92c80, {0x6b55ce0, 0xc00c9ac720}, 0xc009585200, 0x52d1e10)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:468 +0x32c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).SendWithWriteBytes(0xc00ae92c80, {0x6b55ce0?, 0xc00c9ac4e0?}, 0xc009585200)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_send.go:185 +0x6b7
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).SendWithWriteBytes(0xc00810b500, {0x6b55ce0?, 0xc00c9747e0?}, 0xc009585200)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_send.go:193 +0x85c
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Stores).SendWithWriteBytes(0x1?, {0x6b55ce0, 0xc00c9747e0}, 0xc009585200)
    github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/stores.go:202 +0xef
github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc006bde000, {0x6b55ce0?, 0xc00c974420?}, {0xc0076881e0?}, 0xc009585200)
    github.com/cockroachdb/cockroach/pkg/server/node.go:1328 +0x545
github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc006bde000, {0x6b55ce0, 0xc00c9743c0}, 0xc009585200)
    github.com/cockroachdb/cockroach/pkg/server/node.go:1462 +0x2b5
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func1({0x6b55ce0?, 0xc00c9743c0?}, {0x4f20380?, 0xc009585200?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:704 +0x45
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.ServerInterceptor.func12({0x6b55ce0, 0xc00c9743c0}, {0x4f20380, 0xc009585200}, 0xc00df02da0, 0xc011b17308)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:97 +0x4b9
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x6b55ce0?, 0xc00c9743c0?}, {0x4f20380?, 0xc009585200?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x37
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func3({0x6b55ce0, 0xc00c9743c0}, {0x4f20380, 0xc009585200}, 0xc00c9743c0?, 0xc00df02e00)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:169 +0x76
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x6b55ce0?, 0xc00c9743c0?}, {0x4f20380?, 0xc009585200?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x37
github.com/cockroachdb/cockroach/pkg/rpc.kvAuth.unaryInterceptor({0xc002502000?, {{0x4813760?}, {0x6b8c9f0?, 0xc00b19c930?}}}, {0x6b55ce0, 0xc00c9743c0}, {0x4f20380, 0xc009585200}, 0xc00df02da0, 0xc00df02e40)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/auth.go:105 +0x273
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x6b55ce0?, 0xc00c9743c0?}, {0x4f20380?, 0xc009585200?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x37
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1.1({0x6b55ce0?, 0xc00c9743c0?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:136 +0x36
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc002000000, {0x6b55ce0?, 0xc00c9743c0}, {0xc0007cc540?, 0xc019b10908?}, 0xc019b10900)
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:336 +0xd3
github.com/cockroachdb/cockroach/pkg/rpc.NewServerEx.func1({0x6b55ce0?, 0xc00c9743c0?}, {0x4f20380?, 0xc009585200?}, 0x5583a5?, 0xc00ccf60c8?)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:134 +0x8f
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.chainUnaryServerInterceptors.bindUnaryServerInterceptorToHandler.func4({0x6b55ce0?, 0xc00c9743c0?}, {0x4f20380?, 0xc009585200?})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:815 +0x37
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func2({0x6b55ce0?, 0xc00c9743c0?}, {0xd43df6?, 0xc00ccf6058?}, {0x4f20380?, 0xc009585200?}, {0x4dfc5a0?, 0xc00a3c7800}, 0x1?, {0x0, ...})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:714 +0x51
github.com/cockroachdb/cockroach/pkg/rpc.NewContext.ClientInterceptor.func8({0x6b55ce0, 0xc00c9743c0}, {0x5007276, 0x21}, {0x4f20380, 0xc009585200}, {0x4dfc5a0, 0xc00a3c7800}, 0x14659f2?, 0xc004fbbe10, ...)
    github.com/cockroachdb/cockroach/pkg/util/tracing/grpcinterceptor/grpc_interceptor.go:228 +0x47a
github.com/cockroachdb/cockroach/pkg/rpc.getChainUnaryInvoker.func1({0x6b55ce0, 0xc00c9743c0}, {0x5007276, 0x21}, {0x4f20380, 0xc009585200}, {0x4dfc5a0, 0xc00a3c7800}, 0x24?, {0x0, ...})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:899 +0x14e
github.com/cockroachdb/cockroach/pkg/rpc.makeInternalClientAdapter.func3({0x6b55ce0, 0xc00ccf4840}, 0xc0095850e0, {0x0, 0x0, 0x0})
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:784 +0x32f
github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...)
    github.com/cockroachdb/cockroach/pkg/rpc/pkg/rpc/context.go:907
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).sendBatch(0xc00b8031a0, {0x6b55ce0, 0xc00ccf4840}, 0xe7?, {0x6b3a800, 0xc00bf3f7a0?}, 0xc0095850e0)
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:211 +0x1bc
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*grpcTransport).SendNext(0xc00b8031a0, {0x6b55ce0, 0xc00ccf4840}, 0x6b55ce0?)
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/transport.go:189 +0x87
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendToReplicas(0xc012fc2c00, {0x6b55ce0, 0xc00ccf4840}, 0xc010645200?, {0xc00749a730, 0xc00afeec30, 0xc00afeeca0, 0x0, 0x0}, 0x0)
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:2419 +0x11f6
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatch(0xc012fc2c00, {0x6b55ce0?, 0xc00ccf4840}, 0xc010645200, {{0xc0096366f0, 0x14, 0x18}, {0xc00a8859b0, 0x14, 0x18}}, ...)
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1920 +0x7b2
github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*DistSender).sendPartialBatchAsync.func1({0x6b55ce0, 0xc00ccf4840})
    github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/dist_sender.go:1797 +0x147
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x13a
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 3669107
    github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x415
Log preceding fatal error

``` === RUN TestKVNemesisSingleNode_ReproposalChaos test_log_scope.go:170: test logs captured to: /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/logTestKVNemesisSingleNode_ReproposalChaos526563244 test_log_scope.go:81: use -show-logs to present logs inline kvnemesis_test.go:283: seed: 4347807450685509218 kvnemesis_test.go:197: kvnemesis logging to /artifacts/tmp/_tmp/1f42cf5be2fc021646bf9b2daf5eaef3/kvnemesis3461447461 ```

Parameters: TAGS=bazel,gss,deadlock , stress=true

Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

Same failure on other branches

- #115598 kv/kvnemesis: TestKVNemesisSingleNode_ReproposalChaos failed [A-kv-transactions C-bug C-test-failure O-robot T-kv branch-release-23.2]

This test on roachdash | Improve this report!

arulajmani commented 10 months ago

Closing this as a duplicate of https://github.com/cockroachdb/cockroach/issues/115694. We won't see these any longer now that https://github.com/cockroachdb/cockroach/pull/116249 is in.

We've previously determined these aren't release blockers, and once the underlying issues are resolved, validation skipped by https://github.com/cockroachdb/cockroach/pull/116249 can be reverted.