search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.2k stars 3.82k forks source link

Sentry: db.go:817: log.Fatal: pebble: fatal commit error: write ×: There is not enough space on the disk. (1) attached stack trace -- stack trace: | github.com/cockroachdb/pebble.(*DB).applyInter... #131954

Closed cockroach-sentry closed 2 weeks ago

cockroach-sentry commented 1 month ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5952454130/?referrer=webhooks_plugin

Panic Message:

db.go:817: log.Fatal: pebble: fatal commit error: write ×: There is not enough space on the disk.
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/pebble.(*DB).applyInternal
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:817
  | github.com/cockroachdb/pebble.(*DB).Apply
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:752
  | github.com/cockroachdb/pebble.(*DB).Set
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:605
  | github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).put
  |     github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1608
  | github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).PutUnversioned
  |     github.com/cockroachdb/cockroach/pkg/storage/pebble.go:1585
  | github.com/cockroachdb/cockroach/pkg/storage.(*putBuffer).putInlineMeta
  |     github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1385
  | github.com/cockroachdb/cockroach/pkg/storage.mvccPutInternal
  |     github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1805
  | github.com/cockroachdb/cockroach/pkg/storage.mvccPutUsingIter
  |     github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1532
  | github.com/cockroachdb/cockroach/pkg/storage.MVCCBlindPut
  |     github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1476
  | github.com/cockroachdb/cockroach/pkg/storage.MVCCBlindPutProto
  |     github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:860
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.StateLoader.SetRaftReplicaID
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/stateloader.go:187
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvstorage.CreateUninitializedReplica
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvstorage/replica_state.go:155
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).tryGetOrCreateReplica
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_create_replica.go:211
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).getOrCreateReplica
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_create_replica.go:79
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).withReplicaForRequest
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:332
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processRequestQueue
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:589
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:376
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  | runtime.goexit
  |     src/runtime/asm_amd64.s:1594
Wraps: (2) secondary error attachment
  | write ×: There is not enough space on the disk.
  | (1) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/pebble/vfs.(*syncingFile).Write
  |   |     github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go:67
  |   | github.com/cockroachdb/pebble/record.(*LogWriter).flushPending
  |   |     github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:537
  |   | github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop
  |   |     github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:477
  |   | runtime/pprof.Do
  |   |     GOROOT/src/runtime/pprof/runtime.go:40
  |   | github.com/cockroachdb/pebble/record.NewLogWriter.func2
  |   |     github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:360
  |   | runtime.goexit
  |   |     src/runtime/asm_amd64.s:1594
  | Wraps: (2) write ×
  | Wraps: (3) There is not enough space on the disk.
  | Error types: (1) *withstack.withStack (2) *fs.PathError (3) syscall.Errno
Wraps: (3) log.Fatal: pebble: fatal commit error: write ×: There is not enough space on the disk.
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: pebble: fatal commit error: write ×: There is not enough space on the disk.
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
db.go:817: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets): [src/runtime/asm_amd64.s#L1593-L1595](src/runtime/asm_amd64.s#L1593-L1595) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/util/stop/stopper.go#L469-L471](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/util/stop/stopper.go#L469-L471) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L375-L377](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L375-L377) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L588-L590](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L588-L590) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L331-L333](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L331-L333) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/pkg/kv/kvserver/store_create_replica.go#L78-L80](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/pkg/kv/kvserver/store_create_replica.go#L78-L80) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/pkg/kv/kvserver/store_create_replica.go#L210-L212](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/pkg/kv/kvserver/store_create_replica.go#L210-L212) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/kvstorage/replica_state.go#L154-L156](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/kvstorage/replica_state.go#L154-L156) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/logstore/stateloader.go#L186-L188](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/kv/kvserver/logstore/stateloader.go#L186-L188) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/mvcc.go#L859-L861](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/mvcc.go#L859-L861) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/mvcc.go#L1475-L1477](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/mvcc.go#L1475-L1477) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/mvcc.go#L1531-L1533](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/mvcc.go#L1531-L1533) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/mvcc.go#L1804-L1806](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/mvcc.go#L1804-L1806) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/mvcc.go#L1384-L1386](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/mvcc.go#L1384-L1386) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/pebble.go#L1584-L1586](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/pebble.go#L1584-L1586) [https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/pebble.go#L1607-L1609](https://github.com/cockroachdb/cockroach/blob/69a32fafce429e5b75dba8f339ea15922286a0ba/pkg/storage/pebble.go#L1607-L1609) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L604-L606](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L604-L606) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L751-L753](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L751-L753) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L816-L818](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L816-L818) 
src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 376
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processRequestQueue at line 589
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).withReplicaForRequest at line 332
pkg/kv/kvserver/pkg/kv/kvserver/store_create_replica.go in pkg/kv/kvserver.(*Store).getOrCreateReplica at line 79
pkg/kv/kvserver/pkg/kv/kvserver/store_create_replica.go in pkg/kv/kvserver.(*Store).tryGetOrCreateReplica at line 211
pkg/kv/kvserver/kvstorage/replica_state.go in pkg/kv/kvserver/kvstorage.CreateUninitializedReplica at line 155
pkg/kv/kvserver/logstore/stateloader.go in pkg/kv/kvserver/logstore.StateLoader.SetRaftReplicaID at line 187
pkg/storage/mvcc.go in pkg/storage.MVCCBlindPutProto at line 860
pkg/storage/mvcc.go in pkg/storage.MVCCBlindPut at line 1476
pkg/storage/mvcc.go in pkg/storage.mvccPutUsingIter at line 1532
pkg/storage/mvcc.go in pkg/storage.mvccPutInternal at line 1805
pkg/storage/mvcc.go in pkg/storage.(*putBuffer).putInlineMeta at line 1385
pkg/storage/pebble.go in pkg/storage.(*Pebble).PutUnversioned at line 1585
pkg/storage/pebble.go in pkg/storage.(*Pebble).put at line 1608
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).Set at line 605
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).Apply at line 752
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).applyInternal at line 817

Tags

Tag Value
Command server
Environment v23.1.15
Go Version go1.19.13
Platform windows amd64
Distribution CCL
Cockroach Release v23.1.15
Cockroach SHA 69a32fafce429e5b75dba8f339ea15922286a0ba
# of CPUs 8
# of Goroutines 722

Jira issue: CRDB-42761

yuzefovich commented 2 weeks ago

unactionable