search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.21k stars 3.82k forks source link

raftentry: v23.1.13: term regression idx 35: 9 -> 8 #117417

Closed cockroach-sentry closed 2 months ago

cockroach-sentry commented 11 months ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/4816039862/?referrer=webhooks_plugin

Panic Message:

cache.go:384: term regression idx 35: 9 -> 8
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  |     GOROOT/src/runtime/panic.go:884
  | [...repeated from below...]
Wraps: (2) assertion failure
Wraps: (3) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftentry.analyzeEntries
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftentry/cache.go:384
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftentry.(*Cache).Add
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftentry/cache.go:148
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.LoadEntries
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:579
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaRaftStorage).Entries
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go:87
  | go.etcd.io/raft/v3.(*raftLog).slice
  |     go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go:501
  | go.etcd.io/raft/v3.(*raftLog).nextCommittedEnts
  |     go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go:235
  | go.etcd.io/raft/v3.(*RawNode).readyWithoutAccept
  |     go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go:144
  | go.etcd.io/raft/v3.(*RawNode).Ready
  |     go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go:132
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:778
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked.func1
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1997
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:770
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:726
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:395
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  | runtime.goexit
  |     GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (4) term regression idx 35: 9 -> 8
Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
-- report composition:
*errutil.leafError: term regression idx 35: 9 -> 8
cache.go:384: *withstack.withStack (top exception)
*assert.withAssertionFailure
panic.go:884: *withstack.withStack (1)
(check the extra data payloads)
Stacktrace (expand for inline code snippets): [GOROOT/src/runtime/asm_amd64.s#L1593-L1595](GOROOT/src/runtime/asm_amd64.s#L1593-L1595) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/util/stop/stopper.go#L469-L471](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/util/stop/stopper.go#L469-L471) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L725-L727](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L725-L727) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L769-L771](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L769-L771) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1997-L1999](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1997-L1999) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1996-L1998](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1996-L1998) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L777-L779](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L777-L779) [go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go#L131-L133](go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go#L131-L133) [go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go#L143-L145](go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go#L143-L145) [go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go#L234-L236](go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go#L234-L236) [go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go#L500-L502](go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go#L500-L502) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go#L86-L88](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go#L86-L88) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/logstore/logstore.go#L578-L580](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/logstore/logstore.go#L578-L580) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/raftentry/cache.go#L147-L149](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/raftentry/cache.go#L147-L149) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/raftentry/cache.go#L384-L386](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/raftentry/cache.go#L384-L386) [GOROOT/src/runtime/panic.go#L883-L885](GOROOT/src/runtime/panic.go#L883-L885) [GOROOT/src/runtime/asm_amd64.s#L1593-L1595](GOROOT/src/runtime/asm_amd64.s#L1593-L1595) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/util/stop/stopper.go#L469-L471](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/util/stop/stopper.go#L469-L471) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L725-L727](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L725-L727) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L769-L771](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L769-L771) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1997-L1999](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1997-L1999) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1996-L1998](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1996-L1998) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L777-L779](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L777-L779) [go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go#L131-L133](go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go#L131-L133) [go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go#L143-L145](go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go#L143-L145) [go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go#L234-L236](go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go#L234-L236) [go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go#L500-L502](go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go#L500-L502) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go#L86-L88](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go#L86-L88) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/logstore/logstore.go#L578-L580](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/logstore/logstore.go#L578-L580) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/raftentry/cache.go#L147-L149](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/raftentry/cache.go#L147-L149) [https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/raftentry/cache.go#L383-L385](https://github.com/cockroachdb/cockroach/blob/8d065df26e25a762a1abbbf2b1e475456b341a00/pkg/kv/kvserver/raftentry/cache.go#L383-L385) 
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 395
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 646
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 726
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 770
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).withRaftGroupLocked at line 1998
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).withRaftGroupLocked.func1 at line 1997
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2 at line 778
go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go in go.etcd.io/raft/v3.(*RawNode).Ready at line 132
go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go in go.etcd.io/raft/v3.(*RawNode).readyWithoutAccept at line 144
go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go in go.etcd.io/raft/v3.(*raftLog).nextCommittedEnts at line 235
go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go in go.etcd.io/raft/v3.(*raftLog).slice at line 501
pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go in pkg/kv/kvserver.(*replicaRaftStorage).Entries at line 87
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.LoadEntries at line 579
pkg/kv/kvserver/raftentry/cache.go in pkg/kv/kvserver/raftentry.(*Cache).Add at line 148
pkg/kv/kvserver/raftentry/cache.go in pkg/kv/kvserver/raftentry.analyzeEntries at line 385
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 395
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 646
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 726
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 770
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).withRaftGroupLocked at line 1998
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).withRaftGroupLocked.func1 at line 1997
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2 at line 778
go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go in go.etcd.io/raft/v3.(*RawNode).Ready at line 132
go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go in go.etcd.io/raft/v3.(*RawNode).readyWithoutAccept at line 144
go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go in go.etcd.io/raft/v3.(*raftLog).nextCommittedEnts at line 235
go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go in go.etcd.io/raft/v3.(*raftLog).slice at line 501
pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go in pkg/kv/kvserver.(*replicaRaftStorage).Entries at line 87
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.LoadEntries at line 579
pkg/kv/kvserver/raftentry/cache.go in pkg/kv/kvserver/raftentry.(*Cache).Add at line 148
pkg/kv/kvserver/raftentry/cache.go in pkg/kv/kvserver/raftentry.analyzeEntries at line 384

Tags

Tag Value
Command start-single-node
Environment v23.1.13
Go Version go1.19.13
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.13
Cockroach SHA 8d065df26e25a762a1abbbf2b1e475456b341a00
# of CPUs 4
# of Goroutines 199

Jira issue: CRDB-35164

Epic CRDB-39898

blathers-crl[bot] commented 9 months ago

cc @cockroachdb/replication