search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.17k stars 3.82k forks source link

kv/kvserver: TestRaftLogQueue failed #134053

Closed cockroach-teamcity closed 2 weeks ago

cockroach-teamcity commented 3 weeks ago

kv/kvserver.TestRaftLogQueue failed on master @ f9918d8f81a1829df63ac734fd6d21c60141e338:

Fatal error:

panic: KeyComparison of ("\xfa\x00\x00\x18\x03\xd30\xf0\\\x19)\t", "\xfa\x00\x18\x03\xd30\xf0\x04F\xd2\t") = (prefix={3,common=2} cmp=1), ComparePrev gave (prefix={3,common=1} cmp=1)

Stack: 

goroutine 4470 [running]:
github.com/cockroachdb/pebble/sstable/colblk.AssertKeyCompare(0x134851a0, {0xc00af74b80, 0xc, 0x80}, {0xc00bd76f00, 0xb, 0x80}, {0x3, 0x1, 0x1})
    external/com_github_cockroachdb_pebble/sstable/colblk/data_block.go:113 +0x40f
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).evaluatePoint(0xc00e738c08, {{0xc00af74b80?, 0x2?, 0x80?}, 0xc00fadc000?}, 0x3f4)
    external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:477 +0x165
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).AddWithForceObsolete(0xc00e738c08, {{0xc00af74b80?, 0xc?, 0x80?}, 0x62ba8?}, {0xc00fadc000, 0x3f4, 0x26000}, 0x0)
    external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:347 +0x198
github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable(0xc00b7a0e88, {0xecb0200, 0xc00e738c08})
    external/com_github_cockroachdb_pebble/internal/compact/run.go:203 +0xa7c
github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable(0xc00b7a0e88, {0x9, 0x2, {0x0, 0x0, {0x0, 0x0}, 0x0, {0x0, 0x0}, ...}}, ...)
    external/com_github_cockroachdb_pebble/internal/compact/run.go:143 +0x26f
github.com/cockroachdb/pebble.(*DB).compactAndWrite(0xc004ee3688, 0x8, 0xc009bc4908, {0x0, 0x0, 0x0}, 0x7)
    external/com_github_cockroachdb_pebble/compaction.go:2913 +0xdc6
github.com/cockroachdb/pebble.(*DB).runCompaction(0xc004ee3688, 0x8, 0xc009bc4908)
    external/com_github_cockroachdb_pebble/compaction.go:2808 +0xc45
github.com/cockroachdb/pebble.(*DB).flush1(0xc004ee3688)
    external/com_github_cockroachdb_pebble/compaction.go:1448 +0x88d
github.com/cockroachdb/pebble.(*DB).flush.func1({0xec3ad38?, 0xc00a2baed0?})
    external/com_github_cockroachdb_pebble/compaction.go:1183 +0x145
runtime/pprof.Do({0xec3a448, 0x14b4a140}, {{0xc00139cca0?, 0x1f3b6c0?, 0xc004c43a40?}}, 0xc005258fa0)
    GOROOT/src/runtime/pprof/runtime.go:51 +0x118
github.com/cockroachdb/pebble.(*DB).flush(0xc004ee3688)
    external/com_github_cockroachdb_pebble/compaction.go:1176 +0x92
created by github.com/cockroachdb/pebble.(*DB).maybeScheduleFlush in goroutine 38
    external/com_github_cockroachdb_pebble/compaction.go:1092 +0x187
Log preceding fatal error

``` === RUN TestRaftLogQueue test_log_scope.go:165: test logs captured to: outputs.zip/logTestRaftLogQueue2518686210 test_log_scope.go:76: use -show-logs to present logs inline ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/kv

This test on roachdash | Improve this report!

Jira issue: CRDB-43903

arulajmani commented 3 weeks ago

Panic in Pebble. Moving to Storage.

cockroach-teamcity commented 3 weeks ago

kv/kvserver.TestRaftLogQueue failed on master @ 17535c13cfed95db70cd8dfb1ba6a700686f57b1:

Fatal error:

panic: KeyComparison of ("\xfa\x00\x00\x18\x04\"i\xa4\xf6\x17\xc4\t", "\xfa\x00\x18\x04\"i\xa4\x9f\xe3\"\t") = (prefix={3,common=2} cmp=1), ComparePrev gave (prefix={3,common=1} cmp=1)

Stack: 

goroutine 4437 [running]:
github.com/cockroachdb/pebble/sstable/colblk.AssertKeyCompare(0x1348c200, {0xc004655900, 0xc, 0x80}, {0xc004655a00, 0xb, 0x80}, {0x3, 0x1, 0x1})
    external/com_github_cockroachdb_pebble/sstable/colblk/data_block.go:113 +0x40f
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).evaluatePoint(0xc005d03308, {{0xc004655900?, 0x2?, 0x80?}, 0xc00b694000?}, 0x3f4)
    external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:477 +0x165
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).AddWithForceObsolete(0xc005d03308, {{0xc004655900?, 0xc?, 0x80?}, 0x63bd0?}, {0xc00b694000, 0x3f4, 0x26000}, 0x0)
    external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:347 +0x198
github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable(0xc00b656e88, {0xecb5520, 0xc005d03308})
    external/com_github_cockroachdb_pebble/internal/compact/run.go:203 +0xa7c
github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable(0xc00b656e88, {0x9, 0x2, {0x0, 0x0, {0x0, 0x0}, 0x0, {0x0, 0x0}, ...}}, ...)
    external/com_github_cockroachdb_pebble/internal/compact/run.go:143 +0x26f
github.com/cockroachdb/pebble.(*DB).compactAndWrite(0xc002c85b08, 0x8, 0xc002cb2608, {0x0, 0x0, 0x0}, 0x7)
    external/com_github_cockroachdb_pebble/compaction.go:2913 +0xdc6
github.com/cockroachdb/pebble.(*DB).runCompaction(0xc002c85b08, 0x8, 0xc002cb2608)
    external/com_github_cockroachdb_pebble/compaction.go:2808 +0xc45
github.com/cockroachdb/pebble.(*DB).flush1(0xc002c85b08)
    external/com_github_cockroachdb_pebble/compaction.go:1448 +0x88d
github.com/cockroachdb/pebble.(*DB).flush.func1({0xec3ff58?, 0xc00b7fe390?})
    external/com_github_cockroachdb_pebble/compaction.go:1183 +0x145
runtime/pprof.Do({0xec3f668, 0x14b51240}, {{0xc0000b7220?, 0xc004ed9b80?, 0xec8ffd0?}}, 0xc009a9cfa0)
    GOROOT/src/runtime/pprof/runtime.go:51 +0x118
github.com/cockroachdb/pebble.(*DB).flush(0xc002c85b08)
    external/com_github_cockroachdb_pebble/compaction.go:1176 +0x92
created by github.com/cockroachdb/pebble.(*DB).maybeScheduleFlush in goroutine 13
    external/com_github_cockroachdb_pebble/compaction.go:1092 +0x187
Log preceding fatal error

``` === RUN TestRaftLogQueue test_log_scope.go:165: test logs captured to: outputs.zip/logTestRaftLogQueue532571609 test_log_scope.go:76: use -show-logs to present logs inline ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

kv/kvserver.TestRaftLogQueue failed on master @ 6bb6dc96ebf0ee2f23c5c568fa0d421019dc0946:

Fatal error:

panic: KeyComparison of ("\xfa\x00\x00\x18\x04t\xb5\xa3\xf9 \xf6\t", "\xfa\x00\x18\x04t\xb5\xa3\u0089\xe2\t") = (prefix={3,common=2} cmp=1), ComparePrev gave (prefix={3,common=1} cmp=1)

Stack: 

goroutine 4439 [running]:
github.com/cockroachdb/pebble/sstable/colblk.AssertKeyCompare(0x13490200, {0xc0029df600, 0xc, 0x80}, {0xc0029df680, 0xb, 0x80}, {0x3, 0x1, 0x1})
    external/com_github_cockroachdb_pebble/sstable/colblk/data_block.go:113 +0x40f
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).evaluatePoint(0xc0046db908, {{0xc0029df600?, 0x2?, 0x80?}, 0xc008ed4000?}, 0x3f4)
    external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:477 +0x165
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).AddWithForceObsolete(0xc0046db908, {{0xc0029df600?, 0xc?, 0x80?}, 0x636a2?}, {0xc008ed4000, 0x3f4, 0x26000}, 0x0)
    external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:347 +0x198
github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable(0xc008da0e88, {0xecb8080, 0xc0046db908})
    external/com_github_cockroachdb_pebble/internal/compact/run.go:203 +0xa7c
github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable(0xc008da0e88, {0x9, 0x2, {0x0, 0x0, {0x0, 0x0}, 0x0, {0x0, 0x0}, ...}}, ...)
    external/com_github_cockroachdb_pebble/internal/compact/run.go:143 +0x26f
github.com/cockroachdb/pebble.(*DB).compactAndWrite(0xc003ba2008, 0x8, 0xc00a547208, {0x0, 0x0, 0x0}, 0x7)
    external/com_github_cockroachdb_pebble/compaction.go:2913 +0xdc6
github.com/cockroachdb/pebble.(*DB).runCompaction(0xc003ba2008, 0x8, 0xc00a547208)
    external/com_github_cockroachdb_pebble/compaction.go:2808 +0xc45
github.com/cockroachdb/pebble.(*DB).flush1(0xc003ba2008)
    external/com_github_cockroachdb_pebble/compaction.go:1448 +0x88d
github.com/cockroachdb/pebble.(*DB).flush.func1({0xec42ab8?, 0xc00d119410?})
    external/com_github_cockroachdb_pebble/compaction.go:1183 +0x145
runtime/pprof.Do({0xec421c8, 0x14b55280}, {{0xc0014ffb00?, 0x1f3b840?, 0xc00333c780?}}, 0xc008799fa0)
    GOROOT/src/runtime/pprof/runtime.go:51 +0x118
github.com/cockroachdb/pebble.(*DB).flush(0xc003ba2008)
    external/com_github_cockroachdb_pebble/compaction.go:1176 +0x92
created by github.com/cockroachdb/pebble.(*DB).maybeScheduleFlush in goroutine 36
    external/com_github_cockroachdb_pebble/compaction.go:1092 +0x187
Log preceding fatal error

``` === RUN TestRaftLogQueue test_log_scope.go:165: test logs captured to: outputs.zip/logTestRaftLogQueue1921922973 test_log_scope.go:76: use -show-logs to present logs inline ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

kv/kvserver.TestRaftLogQueue failed on master @ 546c489ae32a83b8d312977156cc8f2be93b6a49:

Fatal error:

panic: KeyComparison of ("\xfa\x00\x00\x18\x04\u0096\xa1\xecr\xc0\t", "\xfa\x00\x18\x04\u0096\xa1pP\xc7\t") = (prefix={3,common=2} cmp=1), ComparePrev gave (prefix={3,common=1} cmp=1)

Stack: 

goroutine 4946 [running]:
github.com/cockroachdb/pebble/sstable/colblk.AssertKeyCompare(0x1348f240, {0xc00b613900, 0xc, 0x80}, {0xc00b613980, 0xb, 0x80}, {0x3, 0x1, 0x1})
    external/com_github_cockroachdb_pebble/sstable/colblk/data_block.go:113 +0x40f
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).evaluatePoint(0xc00134f908, {{0xc00b613900?, 0x2?, 0x80?}, 0xc0108b4000?}, 0x3ed)
    external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:477 +0x165
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).AddWithForceObsolete(0xc00134f908, {{0xc00b613900?, 0xc?, 0x80?}, 0x672cb?}, {0xc0108b4000, 0x3ed, 0x28000}, 0x0)
    external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:347 +0x198
github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable(0xc0107e4e88, {0xecb6a60, 0xc00134f908})
    external/com_github_cockroachdb_pebble/internal/compact/run.go:203 +0xa7c
github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable(0xc0107e4e88, {0x9, 0x2, {0x0, 0x0, {0x0, 0x0}, 0x0, {0x0, 0x0}, ...}}, ...)
    external/com_github_cockroachdb_pebble/internal/compact/run.go:143 +0x26f
github.com/cockroachdb/pebble.(*DB).compactAndWrite(0xc002e60d88, 0x8, 0xc00b7a2f08, {0x0, 0x0, 0x0}, 0x7)
    external/com_github_cockroachdb_pebble/compaction.go:2913 +0xdc6
github.com/cockroachdb/pebble.(*DB).runCompaction(0xc002e60d88, 0x8, 0xc00b7a2f08)
    external/com_github_cockroachdb_pebble/compaction.go:2808 +0xc45
github.com/cockroachdb/pebble.(*DB).flush1(0xc002e60d88)
    external/com_github_cockroachdb_pebble/compaction.go:1448 +0x88d
github.com/cockroachdb/pebble.(*DB).flush.func1({0xec41498?, 0xc00d5fefc0?})
    external/com_github_cockroachdb_pebble/compaction.go:1183 +0x145
runtime/pprof.Do({0xec40ba8, 0x14b542c0}, {{0xc0002be820?, 0x1f3bd40?, 0xc00f12bfd0?}}, 0xc00f12bfa0)
    GOROOT/src/runtime/pprof/runtime.go:51 +0x118
github.com/cockroachdb/pebble.(*DB).flush(0xc002e60d88)
    external/com_github_cockroachdb_pebble/compaction.go:1176 +0x92
created by github.com/cockroachdb/pebble.(*DB).maybeScheduleFlush in goroutine 29
    external/com_github_cockroachdb_pebble/compaction.go:1092 +0x187
Log preceding fatal error

``` === RUN TestRaftLogQueue test_log_scope.go:165: test logs captured to: outputs.zip/logTestRaftLogQueue2693294220 test_log_scope.go:76: use -show-logs to present logs inline ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

kv/kvserver.TestRaftLogQueue failed on master @ e8f6410ecd99591fd6103dd8461ea3fb0361f8d6:

Fatal error:

panic: KeyComparison of ("\xfa\x00\x00\x18\x05\x12ۓ\xc7\x1ft\t", "\xfa\x00\x18\x05\x12ۓ\x96\x9e\xee\t") = (prefix={3,common=2} cmp=1), ComparePrev gave (prefix={3,common=1} cmp=1)

Stack: 

goroutine 4439 [running]:
github.com/cockroachdb/pebble/sstable/colblk.AssertKeyCompare(0x13478240, {0xc002119480, 0xc, 0x80}, {0xc002883880, 0xb, 0x80}, {0x3, 0x1, 0x1})
    external/com_github_cockroachdb_pebble/sstable/colblk/data_block.go:113 +0x40f
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).evaluatePoint(0xc00b16c008, {{0xc002119480?, 0x2?, 0x80?}, 0xc00c77c000?}, 0x3ed)
    external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:477 +0x165
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).AddWithForceObsolete(0xc00b16c008, {{0xc002119480?, 0xc?, 0x80?}, 0x6562a?}, {0xc00c77c000, 0x3ed, 0x28000}, 0x0)
    external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:347 +0x198
github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable(0xc00c33ae88, {0xeca4680, 0xc00b16c008})
    external/com_github_cockroachdb_pebble/internal/compact/run.go:203 +0xa7c
github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable(0xc00c33ae88, {0x9, 0x2, {0x0, 0x0, {0x0, 0x0}, 0x0, {0x0, 0x0}, ...}}, ...)
    external/com_github_cockroachdb_pebble/internal/compact/run.go:143 +0x26f
github.com/cockroachdb/pebble.(*DB).compactAndWrite(0xc005130488, 0x8, 0xc009e79b08, {0x0, 0x0, 0x0}, 0x7)
    external/com_github_cockroachdb_pebble/compaction.go:2907 +0xdc6
github.com/cockroachdb/pebble.(*DB).runCompaction(0xc005130488, 0x8, 0xc009e79b08)
    external/com_github_cockroachdb_pebble/compaction.go:2802 +0xc45
github.com/cockroachdb/pebble.(*DB).flush1(0xc005130488)
    external/com_github_cockroachdb_pebble/compaction.go:1442 +0x88d
github.com/cockroachdb/pebble.(*DB).flush.func1({0xec2f1b8?, 0xc0020f16b0?})
    external/com_github_cockroachdb_pebble/compaction.go:1180 +0x145
runtime/pprof.Do({0xec2e8c8, 0x14b3e6c0}, {{0xc001446940?, 0x1f3b260?, 0xc0048f2140?}}, 0xc00e039fa0)
    GOROOT/src/runtime/pprof/runtime.go:51 +0x118
github.com/cockroachdb/pebble.(*DB).flush(0xc005130488)
    external/com_github_cockroachdb_pebble/compaction.go:1173 +0x92
created by github.com/cockroachdb/pebble.(*DB).maybeScheduleFlush in goroutine 56
    external/com_github_cockroachdb_pebble/compaction.go:1089 +0x187
Log preceding fatal error

``` === RUN TestRaftLogQueue test_log_scope.go:165: test logs captured to: outputs.zip/logTestRaftLogQueue1452332452 test_log_scope.go:76: use -show-logs to present logs inline ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

kv/kvserver.TestRaftLogQueue failed on master @ f5fde858b71c965da8068e211e46d2a3d297e876:

Fatal error:

panic: KeyComparison of ("\xfa\x00\x00\x18\x05_e=C\x1d\xfc\t", "\xfa\x00\x18\x05_e=\b\x13\x1a\t") = (prefix={3,common=2} cmp=1), ComparePrev gave (prefix={3,common=1} cmp=1)

Stack: 

goroutine 4333 [running]:
github.com/cockroachdb/pebble/sstable/colblk.AssertKeyCompare(0x135a1240, {0xc004659900, 0xc, 0x80}, {0xc004659f00, 0xb, 0x80}, {0x3, 0x1, 0x1})
    external/com_github_cockroachdb_pebble/sstable/colblk/data_block.go:113 +0x40f
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).evaluatePoint(0xc0083dc008, {{0xc004659900?, 0x2?, 0x80?}, 0xc00a6aa000?}, 0x3ed)
    external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:477 +0x165
github.com/cockroachdb/pebble/sstable.(*RawColumnWriter).AddWithForceObsolete(0xc0083dc008, {{0xc004659900?, 0xc?, 0x80?}, 0x63258?}, {0xc00a6aa000, 0x3ed, 0x28000}, 0x0)
    external/com_github_cockroachdb_pebble/sstable/colblk_writer.go:347 +0x198
github.com/cockroachdb/pebble/internal/compact.(*Runner).writeKeysToTable(0xc00995ae88, {0xed9e360, 0xc0083dc008})
    external/com_github_cockroachdb_pebble/internal/compact/run.go:203 +0xa7c
github.com/cockroachdb/pebble/internal/compact.(*Runner).WriteTable(0xc00995ae88, {0x9, 0x2, {0x0, 0x0, {0x0, 0x0}, 0x0, {0x0, 0x0}, ...}}, ...)
    external/com_github_cockroachdb_pebble/internal/compact/run.go:143 +0x26f
github.com/cockroachdb/pebble.(*DB).compactAndWrite(0xc002c89208, 0x8, 0xc005e0f808, {0x0, 0x0, 0x0}, 0x7)
    external/com_github_cockroachdb_pebble/compaction.go:2907 +0xdc6
github.com/cockroachdb/pebble.(*DB).runCompaction(0xc002c89208, 0x8, 0xc005e0f808)
    external/com_github_cockroachdb_pebble/compaction.go:2802 +0xc45
github.com/cockroachdb/pebble.(*DB).flush1(0xc002c89208)
    external/com_github_cockroachdb_pebble/compaction.go:1442 +0x88d
github.com/cockroachdb/pebble.(*DB).flush.func1({0xed28dd8?, 0xc00ac91230?})
    external/com_github_cockroachdb_pebble/compaction.go:1180 +0x145
runtime/pprof.Do({0xed284e8, 0x14c67740}, {{0xc0000b78a0?, 0x1f3b600?, 0xc002ca1360?}}, 0xc007372fa0)
    GOROOT/src/runtime/pprof/runtime.go:51 +0x118
github.com/cockroachdb/pebble.(*DB).flush(0xc002c89208)
    external/com_github_cockroachdb_pebble/compaction.go:1173 +0x92
created by github.com/cockroachdb/pebble.(*DB).maybeScheduleFlush in goroutine 14
    external/com_github_cockroachdb_pebble/compaction.go:1089 +0x187
Log preceding fatal error

``` === RUN TestRaftLogQueue test_log_scope.go:165: test logs captured to: outputs.zip/logTestRaftLogQueue1970598996 test_log_scope.go:76: use -show-logs to present logs inline ```

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

blathers-crl[bot] commented 2 weeks ago

Based on the specified backports for linked PR #134349, I applied the following new label(s) to this issue: branch-release-24.3. Please adjust the labels as needed to match the branches actually affected by this issue, including adding any known older branches.

:owl: Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.