search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.12k stars 3.81k forks source link

Sentry: pebble.go:1535: log.Fatal: local corruption detected: error initializing data block metadata: column 4 is type ×; not × (1) attached stack trace -- stack trace: | github.com/cockroachdb/... #134777

Open cockroach-sentry opened 2 days ago

cockroach-sentry commented 2 days ago

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/6054700971/?referrer=webhooks_plugin

Panic Message:

pebble.go:1535: log.Fatal: local corruption detected: error initializing data block metadata: column 4 is type ×; not ×
(1) attached stack trace
  -- stack trace:
  | github.com/cockroachdb/cockroach/pkg/storage.(*Pebble).makeMetricEtcEventListener.func1
  |     pkg/storage/pebble.go:1535
  | github.com/cockroachdb/pebble.TeeEventListener.func1
  |     external/com_github_cockroachdb_pebble/event.go:767
  | github.com/cockroachdb/pebble.(*DB).compact.func1
  |     external/com_github_cockroachdb_pebble/compaction.go:2168
  | runtime/pprof.Do
  |     GOROOT/src/runtime/pprof/runtime.go:51
  | github.com/cockroachdb/pebble.(*DB).compact
  |     external/com_github_cockroachdb_pebble/compaction.go:2163
  | runtime.goexit
  |     src/runtime/asm_amd64.s:1695
Wraps: (2) secondary error attachment
  | error initializing data block metadata: column 4 is type ×; not ×
  | (1) forced error mark
  |   | ×
  |   | github.com/cockroachdb/errors/withstack/*withstack.withStack::
  | Wraps: (2) attached stack trace
  |   -- stack trace:
  |   | github.com/cockroachdb/pebble/internal/base.CorruptionErrorf
  |   |     external/com_github_cockroachdb_pebble/internal/base/error.go:30
  |   | github.com/cockroachdb/pebble/sstable/colblk.InitDataBlockMetadata.func1
  |   |     external/com_github_cockroachdb_pebble/sstable/colblk/data_block.go:786
  |   | runtime.gopanic
  |   |     GOROOT/src/runtime/panic.go:770
  |   | github.com/cockroachdb/pebble/sstable/colblk.DecodeColumn[...]
  |   |     external/com_github_cockroachdb_pebble/sstable/colblk/block.go:291
  |   | github.com/cockroachdb/pebble/sstable/colblk.(*BlockDecoder).Uints
  |   |     external/com_github_cockroachdb_pebble/sstable/colblk/block.go:364
  |   | github.com/cockroachdb/pebble/sstable/colblk.(*DataBlockDecoder).Init
  |   |     external/com_github_cockroachdb_pebble/sstable/colblk/data_block.go:888
  |   | github.com/cockroachdb/pebble/sstable/colblk.InitDataBlockMetadata
  |   |     external/com_github_cockroachdb_pebble/sstable/colblk/data_block.go:789
  |   | github.com/cockroachdb/pebble/sstable.(*Reader).initDataBlockMetadata
  |   |     external/com_github_cockroachdb_pebble/sstable/reader.go:418
  |   | github.com/cockroachdb/pebble/sstable.(*Reader).readBlockInternal
  |   |     external/com_github_cockroachdb_pebble/sstable/reader.go:575
  |   | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
  |   |     external/com_github_cockroachdb_pebble/sstable/reader.go:411
  |   | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
  |   |     external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:571
  |   | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).firstInternal
  |   |     external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1206
  |   | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).First
  |   |     external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1189
  |   | github.com/cockroachdb/pebble.(*levelIter).First
  |   |     external/com_github_cockroachdb_pebble/level_iter.go:727
  |   | github.com/cockroachdb/pebble.(*mergingIter).First
  |   |     external/com_github_cockroachdb_pebble/merging_iter.go:1135
  |   | github.com/cockroachdb/pebble/internal/compact.(*Iter).First
  |   |     external/com_github_cockroachdb_pebble/internal/compact/iterator.go:369
  |   | github.com/cockroachdb/pebble/internal/compact.NewRunner
  |   |     external/com_github_cockroachdb_pebble/internal/compact/run.go:119
  |   | github.com/cockroachdb/pebble.(*DB).compactAndWrite
  |   |     external/com_github_cockroachdb_pebble/compaction.go:2902
  |   | github.com/cockroachdb/pebble.(*DB).runCompaction
  |   |     external/com_github_cockroachdb_pebble/compaction.go:2808
  |   | github.com/cockroachdb/pebble.(*DB).compact1
  |   |     external/com_github_cockroachdb_pebble/compaction.go:2252
  |   | github.com/cockroachdb/pebble.(*DB).compact.func1
  |   |     external/com_github_cockroachdb_pebble/compaction.go:2166
  |   | runtime/pprof.Do
  |   |     GOROOT/src/runtime/pprof/runtime.go:51
  |   | github.com/cockroachdb/pebble.(*DB).compact
  |   |     external/com_github_cockroachdb_pebble/compaction.go:2163
  |   | runtime.goexit
  |   |     src/runtime/asm_amd64.s:1695
  | Wraps: (3) secondary error attachment
  |   | column 4 is type ×; not ×
  |   | (1) assertion failure
  |   | Wraps: (2) attached stack trace
  |   |   -- stack trace:
  |   |   | github.com/cockroachdb/pebble/sstable/colblk.DecodeColumn[...]
  |   |   |     external/com_github_cockroachdb_pebble/sstable/colblk/block.go:291
  |   |   | github.com/cockroachdb/pebble/sstable/colblk.(*BlockDecoder).Uints
  |   |   |     external/com_github_cockroachdb_pebble/sstable/colblk/block.go:364
  |   |   | github.com/cockroachdb/pebble/sstable/colblk.(*DataBlockDecoder).Init
  |   |   |     external/com_github_cockroachdb_pebble/sstable/colblk/data_block.go:888
  |   |   | github.com/cockroachdb/pebble/sstable/colblk.InitDataBlockMetadata
  |   |   |     external/com_github_cockroachdb_pebble/sstable/colblk/data_block.go:789
  |   |   | github.com/cockroachdb/pebble/sstable.(*Reader).initDataBlockMetadata
  |   |   |     external/com_github_cockroachdb_pebble/sstable/reader.go:418
  |   |   | github.com/cockroachdb/pebble/sstable.(*Reader).readBlockInternal
  |   |   |     external/com_github_cockroachdb_pebble/sstable/reader.go:575
  |   |   | github.com/cockroachdb/pebble/sstable.(*Reader).readDataBlock
  |   |   |     external/com_github_cockroachdb_pebble/sstable/reader.go:411
  |   |   | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).loadDataBlock
  |   |   |     external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:571
  |   |   | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).firstInternal
  |   |   |     external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1206
  |   |   | github.com/cockroachdb/pebble/sstable.(*singleLevelIterator[...]).First
  |   |   |     external/com_github_cockroachdb_pebble/sstable/reader_iter_single_lvl.go:1189
  |   |   | github.com/cockroachdb/pebble.(*levelIter).First
  |   |   |     external/com_github_cockroachdb_pebble/level_iter.go:727
  |   |   | github.com/cockroachdb/pebble.(*mergingIter).First
  |   |   |     external/com_github_cockroachdb_pebble/merging_iter.go:1135
  |   |   | github.com/cockroachdb/pebble/internal/compact.(*Iter).First
  |   |   |     external/com_github_cockroachdb_pebble/internal/compact/iterator.go:369
  |   |   | github.com/cockroachdb/pebble/internal/compact.NewRunner
  |   |   |     external/com_github_cockroachdb_pebble/internal/compact/run.go:119
  |   |   | github.com/cockroachdb/pebble.(*DB).compactAndWrite
  |   |   |     external/com_github_cockroachdb_pebble/compaction.go:2902
  |   |   | github.com/cockroachdb/pebble.(*DB).runCompaction
  |   |   |     external/com_github_cockroachdb_pebble/compaction.go:2808
  |   |   | github.com/cockroachdb/pebble.(*DB).compact1
  |   |   |     external/com_github_cockroachdb_pebble/compaction.go:2252
  |   |   | github.com/cockroachdb/pebble.(*DB).compact.func1
  |   |   |     external/com_github_cockroachdb_pebble/compaction.go:2166
  |   |   | runtime/pprof.Do
  |   |   |     GOROOT/src/runtime/pprof/runtime.go:51
  |   |   | github.com/cockroachdb/pebble.(*DB).compact
  |   |   |     external/com_github_cockroachdb_pebble/compaction.go:2163
  |   |   | runtime.goexit
  |   |   |     src/runtime/asm_amd64.s:1695
  |   | Wraps: (3) column 4 is type ×; not ×
  |   | Error types: (1) *assert.withAssertionFailure (2) *withstack.withStack (3) *errutil.leafError
  | Wraps: (4) error initializing data block metadata: column 4 is type ×; not ×
  | Error types: (1) *markers.withMark (2) *withstack.withStack (3) *secondary.withSecondaryError (4) *errutil.leafError
Wraps: (3) log.Fatal: local corruption detected: error initializing data block metadata: column 4 is type ×; not ×
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError
-- report composition:
*errutil.leafError: log.Fatal: local corruption detected: error initializing data block metadata: column 4 is type ×; not ×
*secondary.withSecondaryError: details for github.com/cockroachdb/errors/withstack/*withstack.withStack:::
pebble.go:1535: *withstack.withStack (top exception)
Stacktrace (expand for inline code snippets): [src/runtime/asm_amd64.s#L1694-L1696](src/runtime/asm_amd64.s#L1694-L1696) [external/com_github_cockroachdb_pebble/compaction.go#L2162-L2164](external/com_github_cockroachdb_pebble/compaction.go#L2162-L2164) [GOROOT/src/runtime/pprof/runtime.go#L50-L52](GOROOT/src/runtime/pprof/runtime.go#L50-L52) [external/com_github_cockroachdb_pebble/compaction.go#L2167-L2169](external/com_github_cockroachdb_pebble/compaction.go#L2167-L2169) [external/com_github_cockroachdb_pebble/event.go#L766-L768](external/com_github_cockroachdb_pebble/event.go#L766-L768) [pkg/storage/pebble.go#L1534-L1536](pkg/storage/pebble.go#L1534-L1536) 
src/runtime/asm_amd64.s in runtime.goexit at line 1695
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact at line 2163
GOROOT/src/runtime/pprof/runtime.go in runtime/pprof.Do at line 51
external/com_github_cockroachdb_pebble/compaction.go in github.com/cockroachdb/pebble.(*DB).compact.func1 at line 2168
external/com_github_cockroachdb_pebble/event.go in github.com/cockroachdb/pebble.TeeEventListener.func1 at line 767
pkg/storage/pebble.go in pkg/storage.(*Pebble).makeMetricEtcEventListener.func1 at line 1535

Tags

Tag Value
Command start-single-node
Environment v24.3.0-beta.3
Go Version go1.22.5 X:nocoverageredesign
Platform linux amd64
Distribution CCL
Cockroach Release v24.3.0-beta.3
Cockroach SHA c077ebf6e98bcd579481b93c83f14184ab94f2e6
# of CPUs 16
# of Goroutines 47

Jira issue: CRDB-44232

blathers-crl[bot] commented 2 days ago

Hi @cockroach-sentry, please add branch-* labels to identify which branch(es) this C-bug affects.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.