search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.2k stars 3.82k forks source link

Sentry: panic.go:884: panic: × (1) attached stack trace -- stack trace: | runtime.gopanic | GOROOT/src/runtime/panic.go:884 | github.com/cockroachdb/pebble/internal/cache.(*Cache).Free | ... #132436

Closed cockroach-sentry closed 2 weeks ago

cockroach-sentry commented 1 month ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/5984059437/?referrer=webhooks_plugin

Panic Message:

panic.go:884: panic: ×
(1) attached stack trace
  -- stack trace:
  | runtime.gopanic
  |     GOROOT/src/runtime/panic.go:884
  | github.com/cockroachdb/pebble/internal/cache.(*Cache).Free
  |     github.com/cockroachdb/pebble/internal/cache/external/com_github_cockroachdb_pebble/internal/cache/clockpro.go:796
  | github.com/cockroachdb/pebble/sstable.(*Reader).readBlock
  |     github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go:2878
  | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).readBlockWithStats
  |     github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go:632
  | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).loadBlock
  |     github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go:539
  | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).seekGEHelper
  |     github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go:841
  | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).SeekGE
  |     github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go:753
  | github.com/cockroachdb/pebble.(*levelIter).SeekGE
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/level_iter.go:735
  | github.com/cockroachdb/pebble.(*mergingIter).seekGE
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/merging_iter.go:1046
  | github.com/cockroachdb/pebble.(*mergingIter).SeekGE
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/merging_iter.go:1108
  | github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go:1257
  | github.com/cockroachdb/pebble.(*Iterator).SeekGE
  |     github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go:1141
  | github.com/cockroachdb/cockroach/pkg/storage.(*pebbleIterator).SeekGE
  |     github.com/cockroachdb/cockroach/pkg/storage/pebble_iterator.go:335
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftlog.(*Iterator).SeekGE
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftlog/iterator.go:111
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftlog.Visit
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/raftlog/iterator.go:140
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore.LoadEntries
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/logstore/logstore.go:576
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*replicaRaftStorage).Entries
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go:87
  | go.etcd.io/raft/v3.(*raftLog).slice
  |     go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go:501
  | go.etcd.io/raft/v3.(*raftLog).nextCommittedEnts
  |     go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go:235
  | go.etcd.io/raft/v3.(*RawNode).readyWithoutAccept
  |     go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go:144
  | go.etcd.io/raft/v3.(*RawNode).Ready
  |     go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go:132
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:778
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked.func1
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1997
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).withRaftGroupLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:1998
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:770
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go:726
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go:646
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:395
  | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
  |     github.com/cockroachdb/cockroach/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go:302
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  |     github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:470
  | runtime.goexit
  |     GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) panic: ×
Error types: (1) *withstack.withStack (2) *errutil.leafError
-- report composition:
*errutil.leafError: panic: ×
panic.go:884: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets): [GOROOT/src/runtime/asm_amd64.s#L1593-L1595](GOROOT/src/runtime/asm_amd64.s#L1593-L1595) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/util/stop/stopper.go#L469-L471](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/util/stop/stopper.go#L469-L471) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L301-L303) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go#L394-L396) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go#L645-L647) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L725-L727](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L725-L727) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L769-L771](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L769-L771) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1997-L1999](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1997-L1999) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1996-L1998](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L1996-L1998) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L777-L779](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go#L777-L779) [go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go#L131-L133](go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go#L131-L133) [go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go#L143-L145](go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go#L143-L145) [go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go#L234-L236](go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go#L234-L236) [go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go#L500-L502](go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go#L500-L502) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go#L86-L88](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go#L86-L88) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/logstore/logstore.go#L575-L577](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/logstore/logstore.go#L575-L577) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/raftlog/iterator.go#L139-L141](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/raftlog/iterator.go#L139-L141) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/raftlog/iterator.go#L110-L112](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/kv/kvserver/raftlog/iterator.go#L110-L112) [https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/storage/pebble_iterator.go#L334-L336](https://github.com/cockroachdb/cockroach/blob/b95f2225b09cf25f9f2be31ef6a6f65c73a1b081/pkg/storage/pebble_iterator.go#L334-L336) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go#L1140-L1142](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go#L1140-L1142) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go#L1256-L1258](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go#L1256-L1258) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/merging_iter.go#L1107-L1109](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/merging_iter.go#L1107-L1109) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/merging_iter.go#L1045-L1047](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/merging_iter.go#L1045-L1047) [github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/level_iter.go#L734-L736](github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/level_iter.go#L734-L736) [github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go#L752-L754](github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go#L752-L754) [github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go#L840-L842](github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go#L840-L842) [github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go#L538-L540](github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go#L538-L540) [github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go#L631-L633](github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go#L631-L633) [github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go#L2877-L2879](github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go#L2877-L2879) [github.com/cockroachdb/pebble/internal/cache/external/com_github_cockroachdb_pebble/internal/cache/clockpro.go#L795-L797](github.com/cockroachdb/pebble/internal/cache/external/com_github_cockroachdb_pebble/internal/cache/clockpro.go#L795-L797) [GOROOT/src/runtime/panic.go#L883-L885](GOROOT/src/runtime/panic.go#L883-L885) 
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 470
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftScheduler).Start.func2 at line 302
pkg/kv/kvserver/pkg/kv/kvserver/scheduler.go in pkg/kv/kvserver.(*raftSchedulerShard).worker at line 395
pkg/kv/kvserver/pkg/kv/kvserver/store_raft.go in pkg/kv/kvserver.(*Store).processReady at line 646
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReady at line 726
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked at line 770
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).withRaftGroupLocked at line 1998
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).withRaftGroupLocked.func1 at line 1997
pkg/kv/kvserver/pkg/kv/kvserver/replica_raft.go in pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked.func2 at line 778
go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go in go.etcd.io/raft/v3.(*RawNode).Ready at line 132
go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/rawnode.go in go.etcd.io/raft/v3.(*RawNode).readyWithoutAccept at line 144
go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go in go.etcd.io/raft/v3.(*raftLog).nextCommittedEnts at line 235
go.etcd.io/raft/v3/external/io_etcd_go_raft_v3/log.go in go.etcd.io/raft/v3.(*raftLog).slice at line 501
pkg/kv/kvserver/pkg/kv/kvserver/replica_raftstorage.go in pkg/kv/kvserver.(*replicaRaftStorage).Entries at line 87
pkg/kv/kvserver/logstore/logstore.go in pkg/kv/kvserver/logstore.LoadEntries at line 576
pkg/kv/kvserver/raftlog/iterator.go in pkg/kv/kvserver/raftlog.Visit at line 140
pkg/kv/kvserver/raftlog/iterator.go in pkg/kv/kvserver/raftlog.(*Iterator).SeekGE at line 111
pkg/storage/pebble_iterator.go in pkg/storage.(*pebbleIterator).SeekGE at line 335
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).SeekGE at line 1141
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/iterator.go in github.com/cockroachdb/pebble.(*Iterator).SeekGEWithLimit at line 1257
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).SeekGE at line 1108
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/merging_iter.go in github.com/cockroachdb/pebble.(*mergingIter).seekGE at line 1046
github.com/cockroachdb/pebble/external/com_github_cockroachdb_pebble/level_iter.go in github.com/cockroachdb/pebble.(*levelIter).SeekGE at line 735
github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).SeekGE at line 753
github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).seekGEHelper at line 841
github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).loadBlock at line 539
github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).readBlockWithStats at line 632
github.com/cockroachdb/pebble/sstable/external/com_github_cockroachdb_pebble/sstable/reader.go in github.com/cockroachdb/pebble/sstable.(*Reader).readBlock at line 2878
github.com/cockroachdb/pebble/internal/cache/external/com_github_cockroachdb_pebble/internal/cache/clockpro.go in github.com/cockroachdb/pebble/internal/cache.(*Cache).Free at line 796
GOROOT/src/runtime/panic.go in runtime.gopanic at line 884

Tags

Tag Value
Command server
Environment v23.1.14
Go Version go1.19.13
Platform linux amd64
Distribution CCL
Cockroach Release v23.1.14
Cockroach SHA b95f2225b09cf25f9f2be31ef6a6f65c73a1b081
# of CPUs 32
# of Goroutines 773

Jira issue: CRDB-43065

yuzefovich commented 2 weeks ago

unactionable