search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.18k stars 3.81k forks source link

Sentry: db.go:857: log.Fatal: pebble: fatal commit error: no data available (1) attached stack trace -- stack trace: | github.com/cockroachdb/pebble.(*DB).applyInternal | github.com/cockroachdb... #131494

Closed cockroach-sentry closed 1 week ago

cockroach-sentry commented 2 months ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5915449509/?referrer=webhooks_plugin

Panic Message:

db.go:857: log.Fatal: pebble: fatal commit error: no data available
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/pebble.(*DB).applyInternal
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:857
  | github.com/cockroachdb/pebble.(*DB).Apply
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:780
  | github.com/cockroachdb/pebble.(*Batch).Commit
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go:1329
  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).Commit
  |     github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:617
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:279
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).StoreEntries
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:150
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1026
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:744
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:689
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:418
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:321
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484
  | runtime.goexit
  |     src/runtime/asm_amd64.s:1650
Wraps: (2) secondary error attachment
  | no data available
  | (1) no data available
  | Error types: (1) syscall.Errno
Wraps: (3) log.Fatal: pebble: fatal commit error: no data available
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: pebble: fatal commit error: no data available
*secondary.withSecondaryError
db.go:857: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets): [src/runtime/asm_amd64.s#L1649-L1651](src/runtime/asm_amd64.s#L1649-L1651) [https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/util/stop/stopper.go#L483-L485](https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/util/stop/stopper.go#L483-L485) [https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L320-L322](https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L320-L322) [https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L417-L419](https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L417-L419) [https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L688-L690](https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L688-L690) [https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L743-L745](https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L743-L745) [https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1025-L1027](https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1025-L1027) [https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/logstore/logstore.go#L149-L151](https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/logstore/logstore.go#L149-L151) [https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/logstore/logstore.go#L278-L280](https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/kv/kvserver/logstore/logstore.go#L278-L280) [https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/storage/pebble_batch.go#L616-L618](https://github.com/cockroachdb/cockroach/blob/4634a65a4891dcca4fc56916481d56721954d813/pkg/storage/pebble_batch.go#L616-L618) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go#L1328-L1330](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go#L1328-L1330) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L779-L781](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L779-L781) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L856-L858](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L856-L858) 
src/runtime/asm_amd64.s in runtime.goexit at line 1650
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 484
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 321
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 418
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 689
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 744
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 1026
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).StoreEntries at line 150
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch at line 279
pkg/storage/pebble_batch.go in pkg/storage.(*pebbleBatch).Commit at line 617
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go in github.com/cockroachdb/pebble.(*Batch).Commit at line 1329
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).Apply at line 780
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).applyInternal at line 857

Tags

Tag Value
Command server
Environment v23.2.2
Go Version go1.21.5 X:nocoverageredesign
Platform linux amd64
Distribution CCL
Cockroach Release v23.2.2
Cockroach SHA 4634a65a4891dcca4fc56916481d56721954d813
# of CPUs 64
# of Goroutines 94

Jira issue: CRDB-42571

yuzefovich commented 1 week ago

unactionable