search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.19k stars 3.82k forks source link

Sentry: syncing_file.go:67: write ×: There is not enough space on the disk. (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | github.com/cockroac... #129861

Closed cockroach-sentry closed 2 weeks ago

cockroach-sentry commented 3 months ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5770942861/?referrer=webhooks_plugin

Panic Message:

syncing_file.go:67: write ×: There is not enough space on the disk.
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  |     GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/pebble.(*DB).commitWrite
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:925
  | github.com/cockroachdb/pebble.(*commitPipeline).prepare
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/commit.go:461
  | github.com/cockroachdb/pebble.(*commitPipeline).Commit
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/commit.go:310
  | github.com/cockroachdb/pebble.(*DB).applyInternal
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:814
  | github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go:771
  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).CommitNoSyncWait
  |     github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:577
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:252
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).StoreEntries
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:144
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:979
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:725
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:395
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
  | [...repeated from below...]
Wraps: (2) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/pebble/vfs.(*syncingFile).Write
  |     github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go:67
  | github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock
  |     github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:562
  | github.com/cockroachdb/pebble/record.(*LogWriter).flushPending
  |     github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:531
  | github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop
  |     github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:477
  | runtime/pprof.Do
  |     GOROOT/src/runtime/pprof/runtime.go:40
  | github.com/cockroachdb/pebble/record.NewLogWriter.func2
  |     github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go:360
  | runtime.goexit
  |     src/runtime/asm_amd64.s:1594
Wraps: (3) write ×
Wraps: (4) There is not enough space on the disk.
Error types: (1) *withstack.withStack (2) *withstack.withStack (3) *fs.PathError (4) syscall.Errno
-- report composition:
syscall.Errno
*fs.PathError
syncing_file.go:67: *withstack.withStack (top exception)
panic.go:884: *withstack.withStack (1)
(check the extra data payloads)
Stacktrace (expand for inline code snippets): [src/runtime/asm_amd64.s#L1593-L1595](src/runtime/asm_amd64.s#L1593-L1595) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/util/stop/stopper.go#L469-L471](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/util/stop/stopper.go#L469-L471) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L724-L726](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L724-L726) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L978-L980](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L978-L980) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/logstore/logstore.go#L143-L145](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/logstore/logstore.go#L143-L145) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/logstore/logstore.go#L251-L253](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/kv/kvserver/logstore/logstore.go#L251-L253) [https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/storage/pebble_batch.go#L576-L578](https://github.com/cockroachdb/cockroach/blob/62ad175cbc0e9041d993ea1cc911e07d279671f5/pkg/storage/pebble_batch.go#L576-L578) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L770-L772](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L770-L772) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L813-L815](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L813-L815) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/commit.go#L309-L311](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/commit.go#L309-L311) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/commit.go#L460-L462](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/commit.go#L460-L462) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L924-L926](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go#L924-L926) [GOROOT/src/runtime/panic.go#L883-L885](GOROOT/src/runtime/panic.go#L883-L885) [src/runtime/asm_amd64.s#L1593-L1595](src/runtime/asm_amd64.s#L1593-L1595) [github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L359-L361](github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L359-L361) [GOROOT/src/runtime/pprof/runtime.go#L39-L41](GOROOT/src/runtime/pprof/runtime.go#L39-L41) [github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L476-L478](github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L476-L478) [github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L530-L532](github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L530-L532) [github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L561-L563](github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go#L561-L563) [github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go#L66-L68](github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go#L66-L68) 
src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 395
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 646
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 725
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 979
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).StoreEntries at line 144
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch at line 252
pkg/storage/pebble_batch.go in pkg/storage.(*pebbleBatch).CommitNoSyncWait at line 577
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).ApplyNoSyncWait at line 771
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).applyInternal at line 814
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/commit.go in github.com/cockroachdb/pebble.(*commitPipeline).Commit at line 310
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/commit.go in github.com/cockroachdb/pebble.(*commitPipeline).prepare at line 461
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/db.go in github.com/cockroachdb/pebble.(*DB).commitWrite at line 925
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
src/runtime/asm_amd64.s in runtime.goexit at line 1594
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.NewLogWriter.func2 at line 360
GOROOT/src/runtime/pprof/runtime.go in runtime/pprof.Do at line 40
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushLoop at line 477
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushPending at line 531
github.com/cockroachdb/pebble/record/external/com_github_cockroachdb_pebble/record/log_writer.go in github.com/cockroachdb/pebble/record.(*LogWriter).flushBlock at line 562
github.com/cockroachdb/pebble/vfs/external/com_github_cockroachdb_pebble/vfs/syncing_file.go in github.com/cockroachdb/pebble/vfs.(*syncingFile).Write at line 67

Tags

Tag Value
Command start-single-node
Environment v23.1.11
Go Version go1.19.10
Platform windows amd64
Distribution CCL
Cockroach Release v23.1.11
Cockroach SHA 62ad175cbc0e9041d993ea1cc911e07d279671f5
# of CPUs 6
# of Goroutines 368

Jira issue: CRDB-41742

yuzefovich commented 2 weeks ago

unactionable