search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.18k stars 3.81k forks source link

Sentry: panic.go:884: panic: × (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).Clos... #128256

Closed cockroach-sentry closed 1 week ago

cockroach-sentry commented 3 months ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5672126687/?referrer=webhooks_plugin

Panic Message:

panic.go:884: panic: ×
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  |     GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).Close
  |     github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:123
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch.func1
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:164
  | runtime.gopanic
  |     GOROOT/src/runtime/panic.go:884
  | runtime.panicmem
  |     GOROOT/src/runtime/panic.go:260
  | runtime.sigpanic
  |     GOROOT/src/runtime/signal_unix.go:839
  | github.com/cockroachdb/pebble.(*Batch).prepareDeferredKeyValueRecord
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go:519
  | github.com/cockroachdb/pebble.(*Batch).SetDeferred
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go:623
  | github.com/cockroachdb/pebble.(*Batch).Set
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go:605
  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).put
  |     github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:535
  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleBatch).PutUnversioned
  |     github.com/cockroachdb/cockroach/pkg/storage/pebble_batch.go:507
  | github.com/cockroachdb/cockroach/pkg/storage.(*putBuffer).putInlineMeta
  |     github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1385
  | github.com/cockroachdb/cockroach/pkg/storage.mvccPutInternal
  |     github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1805
  | github.com/cockroachdb/cockroach/pkg/storage.mvccPutUsingIter
  |     github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1532
  | github.com/cockroachdb/cockroach/pkg/storage.MVCCBlindPut
  |     github.com/cockroachdb/cockroach/pkg/storage/mvcc.go:1476
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.logAppend
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:401
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:182
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.(*LogStore).StoreEntries
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:146
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:988
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:732
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:395
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  | runtime.goexit
  |     GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) panic: ×
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: panic: ×
panic.go:884: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets): [GOROOT/src/runtime/asm_amd64.s#L1593-L1595](GOROOT/src/runtime/asm_amd64.s#L1593-L1595) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/util/stop/stopper.go#L469-L471](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/util/stop/stopper.go#L469-L471) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L731-L733](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L731-L733) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L987-L989](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L987-L989) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/logstore/logstore.go#L145-L147](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/logstore/logstore.go#L145-L147) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/logstore/logstore.go#L181-L183](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/logstore/logstore.go#L181-L183) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/logstore/logstore.go#L400-L402](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/logstore/logstore.go#L400-L402) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/mvcc.go#L1475-L1477](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/mvcc.go#L1475-L1477) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/mvcc.go#L1531-L1533](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/mvcc.go#L1531-L1533) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/mvcc.go#L1804-L1806](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/mvcc.go#L1804-L1806) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/mvcc.go#L1384-L1386](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/mvcc.go#L1384-L1386) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/pebble_batch.go#L506-L508](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/pebble_batch.go#L506-L508) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/pebble_batch.go#L534-L536](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/pebble_batch.go#L534-L536) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go#L604-L606](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go#L604-L606) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go#L622-L624](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go#L622-L624) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go#L518-L520](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go#L518-L520) [GOROOT/src/runtime/signal_unix.go#L838-L840](GOROOT/src/runtime/signal_unix.go#L838-L840) [GOROOT/src/runtime/panic.go#L259-L261](GOROOT/src/runtime/panic.go#L259-L261) [GOROOT/src/runtime/panic.go#L883-L885](GOROOT/src/runtime/panic.go#L883-L885) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/logstore/logstore.go#L163-L165](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/kv/kvserver/logstore/logstore.go#L163-L165) [https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/pebble_batch.go#L122-L124](https://github.com/cockroachdb/cockroach/blob/fceb4cf5c378bfb53b3cf12c672716168192f1fe/pkg/storage/pebble_batch.go#L122-L124) [GOROOT/src/runtime/panic.go#L883-L885](GOROOT/src/runtime/panic.go#L883-L885) 
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 395
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 646
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 732
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 988
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).StoreEntries at line 146
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch at line 182
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.logAppend at line 401
pkg/storage/mvcc.go in pkg/storage.MVCCBlindPut at line 1476
pkg/storage/mvcc.go in pkg/storage.mvccPutUsingIter at line 1532
pkg/storage/mvcc.go in pkg/storage.mvccPutInternal at line 1805
pkg/storage/mvcc.go in pkg/storage.(*putBuffer).putInlineMeta at line 1385
pkg/storage/pebble_batch.go in pkg/storage.(*pebbleBatch).PutUnversioned at line 507
pkg/storage/pebble_batch.go in pkg/storage.(*pebbleBatch).put at line 535
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go in github.com/cockroachdb/pebble.(*Batch).Set at line 605
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go in github.com/cockroachdb/pebble.(*Batch).SetDeferred at line 623
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/batch.go in github.com/cockroachdb/pebble.(*Batch).prepareDeferredKeyValueRecord at line 519
GOROOT/src/runtime/signal_unix.go in runtime.sigpanic at line 839
GOROOT/src/runtime/panic.go in runtime.panicmem at line 260
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.(*LogStore).storeEntriesAndCommitBatch.func1 at line 164
pkg/storage/pebble_batch.go in pkg/storage.(*pebbleBatch).Close at line 123
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884

Tags

Tag Value
Command server
Environment v23.1.21
Go Version go1.19.13
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.21
Cockroach SHA fceb4cf5c378bfb53b3cf12c672716168192f1fe
# of CPUs 16
# of Goroutines 3347

Jira issue: CRDB-40883

yuzefovich commented 1 week ago

unactionable