search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.15k stars 3.81k forks source link

kvserver: v23.1.22: zero-value ReplicatedEvalResult passed to handleNonTrivialReplicatedEvalResult #126576

Open cockroach-sentry opened 4 months ago

cockroach-sentry commented 4 months ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5564615913/?referrer=webhooks_plugin

Panic Message:

replica_application_state_machine.go:277: log.Fatal: zero-value ReplicatedEvalResult passed to handleNonTrivialReplicatedEvalResult
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:277
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go:187
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.mapCheckedCmdIter
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/cmd.go:210
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).applyOneBatch
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:295
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/apply/task.go:251
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1021
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:732
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:395
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  | runtime.goexit
  |     GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) log.Fatal: zero-value ReplicatedEvalResult passed to handleNonTrivialReplicatedEvalResult
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: zero-value ReplicatedEvalResult passed to handleNonTrivialReplicatedEvalResult
replica_application_state_machine.go:277: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets): [GOROOT/src/runtime/asm_amd64.s#L1593-L1595](GOROOT/src/runtime/asm_amd64.s#L1593-L1595) [https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/util/stop/stopper.go#L469-L471](https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/util/stop/stopper.go#L469-L471) [https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303](https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303) [https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396](https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396) [https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647](https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647) [https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L731-L733](https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L731-L733) [https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1020-L1022](https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1020-L1022) [https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/apply/task.go#L250-L252](https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/apply/task.go#L250-L252) [https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/apply/task.go#L294-L296](https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/apply/task.go#L294-L296) [https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/apply/cmd.go#L209-L211](https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/apply/cmd.go#L209-L211) [https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L186-L188](https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L186-L188) [https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L276-L278](https://github.com/cockroachdb/cockroach/blob/6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7/pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go#L276-L278) 
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 395
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 646
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 732
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1021
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).ApplyCommittedEntries at line 251
pkg/kv/kvserver/apply/task.go in pkg/kv/kvserver/apply.(*Task).applyOneBatch at line 295
pkg/kv/kvserver/apply/cmd.go in pkg/kv/kvserver/apply.mapCheckedCmdIter at line 210
pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaStateMachine).ApplySideEffects at line 187
pkg/kv/kvserver/pkg/kv/kvserver/replica_application_state_machine.go in pkg/kv/kvserver.(*replicaStateMachine).handleNonTrivialReplicatedEvalResult at line 277

Tags

Tag Value
Command server
Environment v23.1.22
Go Version go1.19.13
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.22
Cockroach SHA 6ed4ef16634f683adfb7d77b4ebf414e0c1e42a7
# of CPUs 128
# of Goroutines 1885

Jira issue: CRDB-39984

blathers-crl[bot] commented 4 months ago

Hi @cockroach-sentry, please add branch-* labels to identify which branch(es) this C-bug affects.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.

andrewbaptist commented 4 months ago

@pav-kv you may want to take a look at this when you get a chance. I tried reasoning through the paths to get here, but it wasn't clear that there is one. This seems like it might be related to #126432 since they happended on the same cluster (although 3 days apart) https://cockroach-labs.sentry.io/issues/?limit=5&project=164528&query=cluster%3A829dde95-f0fd-4ed4-9880-c85d3b193e0d&referrer=issue-list&sort=freq&statsPeriod=90d

Of note is that there was a replica inconsistency found on r8977 on this same cluster about 6 days after these issues. Unfortunately these first two reports don't specify which range this was on.