search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.18k stars 3.82k forks source link

Sentry: raft.go:1971: log.Fatal: 6 state.commit 20983 is out of range [20967, 20982] (1) attached stack trace -- stack trace: | github.com/cockroachdb/cockroach/pkg/raft.(*raft).loadState | git... #136242

Open cockroach-sentry opened 10 hours ago

cockroach-sentry commented 10 hours ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6092546358/?referrer=webhooks_plugin

Panic Message:

raft.go:1971: log.Fatal: 6 state.commit 20983 is out of range [20967, 20982]
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/raft.(*raft).loadState
  |     github.com/cockroachdb/cockroach/pkg/raft/raft.go:1971
  | github.com/cockroachdb/cockroach/pkg/raft.newRaft
  |     github.com/cockroachdb/cockroach/pkg/raft/raft.go:441
  | github.com/cockroachdb/cockroach/pkg/raft.NewRawNode
  |     github.com/cockroachdb/cockroach/pkg/raft/rawnode.go:55
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).initRaftGroupRaftMuLockedReplicaMuLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go:295
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).initRaftMuLockedReplicaMuLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go:270
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.newInitializedReplica
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go:81
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).Start
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store.go:2207
  | github.com/cockroachdb/cockroach/pkg/server.(*Node).start.func1
  |     github.com/cockroachdb/cockroach/pkg/server/node.go:674
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:485
  | runtime.goexit
  |     src/runtime/asm_amd64.s:1695
Wraps: (2) log.Fatal: 6 state.commit 20983 is out of range [20967, 20982]
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: 6 state.commit 20983 is out of range [20967, 20982]
raft.go:1971: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets): [src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696) [https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/util/stop/stopper.go#L484-L486](https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/util/stop/stopper.go#L484-L486) [https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/server/node.go#L673-L675](https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/server/node.go#L673-L675) [https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/kv/kvserver/pkg/kv/kvserver/store.go#L2206-L2208](https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/kv/kvserver/pkg/kv/kvserver/store.go#L2206-L2208) [https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go#L80-L82](https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go#L80-L82) [https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go#L269-L271](https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go#L269-L271) [https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go#L294-L296](https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go#L294-L296) [https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/raft/rawnode.go#L54-L56](https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/raft/rawnode.go#L54-L56) [https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/raft/raft.go#L440-L442](https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/raft/raft.go#L440-L442) [https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/raft/raft.go#L1970-L1972](https://github.com/cockroachdb/cockroach/blob/8140b8e294fb16a293d0aae662bcf5c1d488c852/pkg/raft/raft.go#L1970-L1972) 
src/runtime/asm_amd64.s in runtime.goexit at line 1695
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 485
pkg/server/node.go in pkg/server.(*Node).start.func1 at line 674
pkg/kv/kvserver/pkg/kv/kvserver/store.go in pkg/kv/kvserver.(*Store).Start at line 2207
pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go in pkg/kv/kvserver.newInitializedReplica at line 81
pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go in pkg/kv/kvserver.(*Replica).initRaftMuLockedReplicaMuLocked at line 270
pkg/kv/kvserver/pkg/kv/kvserver/replica_init.go in pkg/kv/kvserver.(*Replica).initRaftGroupRaftMuLockedReplicaMuLocked at line 295
pkg/raft/rawnode.go in pkg/raft.NewRawNode at line 55
pkg/raft/raft.go in pkg/raft.newRaft at line 441
pkg/raft/raft.go in pkg/raft.(*raft).loadState at line 1971

Tags

Tag Value
Command server
Environment v24.2.4
Go Version go1.22.5 X:nocoverageredesign
Platform linux amd64
Distribution CCL
Cockroach Release v24.2.4
Cockroach SHA 8140b8e294fb16a293d0aae662bcf5c1d488c852
# of CPUs 8
# of Goroutines 391

Jira issue: CRDB-44938

blathers-crl[bot] commented 10 hours ago

CC'ing via the CODEOWNERS-based sentry heuristic:

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.