search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.19k stars 3.82k forks source link

Sentry: replica_proposal_buf.go:445: log.Fatal: unexpected error: request writing below tracked lower bound: wts: 1730396060.356942579,2 < lb: 1730396070.334794991,0; ba: Put [/Table/39/2/×/×/×,/Mi... #133985

Closed cockroach-sentry closed 2 weeks ago

cockroach-sentry commented 4 weeks ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6034109184/?referrer=webhooks_plugin

Panic Message:

replica_proposal_buf.go:445: log.Fatal: unexpected error: request writing below tracked lower bound: wts: 1730396060.356942579,2 < lb: 1730396070.334794991,0; ba: Put [/Table/39/2/×/×/×,/Min), [txn: 38e945e9]; lease: ×.
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:445
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked.func1
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1994
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1995
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:769
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:725
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:395
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  | runtime.goexit
  |     GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) secondary error attachment
  | unexpected error: request writing below tracked lower bound: wts: 1730396060.356942579,2 < lb: 1730396070.334794991,0; ba: Put [/Table/39/2/×/×/×,/Min), [txn: 38e945e9]; lease: ×.
  | (1) ×
  | Wraps: (2) issue #72428
  | Wraps: (3) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/cockroach/pkg/util/errorutil.UnexpectedWithIssueErrorf
  |   |     github.com/cockroachdb/cockroach/pkg/util/errorutil/error.go:29
  |   | [...repeated from below...]
  | Wraps: (4) unexpected error
  | Wraps: (5) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/cockroach/pkg/util/errorutil.UnexpectedWithIssueErrorf
  |   |     github.com/cockroachdb/cockroach/pkg/util/errorutil/error.go:28
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup
  |   |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go:445
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2
  |   |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:772
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked.func1
  |   |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1994
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked
  |   |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1995
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  |   |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:769
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
  |   |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:725
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
  |   |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  |   |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:395
  |   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  |   |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
  |   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |   |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  |   | runtime.goexit
  |   |     GOROOT/src/runtime/asm_amd64.s:1594
  | Wraps: (6) request writing below tracked lower bound: wts: 1730396060.356942579,2 < lb: 1730396070.334794991,0; ba: Put [/Table/39/2/×/×/×,/Min), [txn: 38e945e9]; lease: ×.
  | Error types: (1) *hintdetail.withHint (2) *safedetails.withSafeDetails (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Wraps: (3) log.Fatal: unexpected error: request writing below tracked lower bound: wts: 1730396060.356942579,2 < lb: 1730396070.334794991,0; ba: Put [/Table/39/2/×/×/×,/Min), [txn: 38e945e9]; lease: ×.
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: unexpected error: request writing below tracked lower bound: wts: 1730396060.356942579,2 < lb: 1730396070.334794991,0; ba: Put [/Table/39/2/×/×/×,/Min), [txn: 38e945e9]; lease: ×.
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/safedetails/*safedetails.withSafeDetails:::
replica_proposal_buf.go:445: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets): [GOROOT/src/runtime/asm_amd64.s#L1593-L1595](GOROOT/src/runtime/asm_amd64.s#L1593-L1595) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/util/stop/stopper.go#L469-L471](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/util/stop/stopper.go#L469-L471) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L724-L726](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L724-L726) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L768-L770](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L768-L770) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1994-L1996](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1994-L1996) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1993-L1995](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1993-L1995) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L771-L773](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L771-L773) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go#L444-L446](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go#L444-L446) 
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 395
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 646
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 725
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 769
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).withRaftGroupLocked at line 1995
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).withRaftGroupLocked.func1 at line 1994
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2 at line 772
pkg/kv/kvserver/pkg/kv/kvserver/replica_proposal_buf.go in pkg/kv/kvserver.(*propBuf).FlushLockedWithRaftGroup at line 445

Tags

Tag Value
Command server
Environment v23.1.11
Go Version go1.19.10
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.11
Cockroach SHA 62ad175cbc0e9041d993ea1cc911e07d279671f5
# of CPUs 7
# of Goroutines 383

Jira issue: CRDB-43853

yuzefovich commented 2 weeks ago

unactionable