search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.13k stars 3.81k forks source link

sql: TestRaceWithIndexBackfillMerge failed #135355

Closed cockroach-teamcity closed 2 hours ago

cockroach-teamcity commented 4 hours ago

sql.TestRaceWithIndexBackfillMerge failed with artifacts on master @ 6f00e544ae1f8bf0fcdbcee607376da00353020b:

*   |   pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go:784
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked
*   |   pkg/kv/kvserver/replica_raft.go:1017
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady
*   |   pkg/kv/kvserver/replica_raft.go:836
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady
*   |   pkg/kv/kvserver/store_raft.go:682
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker
*   |   pkg/kv/kvserver/scheduler.go:419
*   | github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2
*   |   pkg/kv/kvserver/scheduler.go:319
*   | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
*   |   pkg/util/stop/stopper.go:498
*   | runtime.goexit
*   |   src/runtime/asm_arm64.s:1222
* Wraps: (4) empty send-queue with non-zero precise size: before=+58 KiB after=+39 KiB delta=-19 KiB [queue_len=0, queue_approx_size=+39 KiB]
* Error types: (1) *withstack.withStack (2) *assert.withAssertionFailure (3) *withstack.withStack (4) *errutil.leafError
*
panic: empty send-queue with non-zero precise size: before=+58 KiB after=+39 KiB delta=-19 KiB [queue_len=0, queue_approx_size=+39 KiB] [recovered]
    panic: empty send-queue with non-zero precise size: before=+58 KiB after=+39 KiB delta=-19 KiB [queue_len=0, queue_approx_size=+39 KiB]

goroutine 58920 [running]:
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).recover(0x400556f8e8?, {0x83ede40, 0x400e5b2de0})
    pkg/util/stop/stopper.go:226 +0x68
panic({0x6322700?, 0x400b0deeb0?})
    GOROOT/src/runtime/panic.go:770 +0x124
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2.(*replicaSendStream).applySendQueuePreciseSizeDeltaRaftMuAndStreamLocked(0xca607c0?, {0x83ede40?, 0x400b6cfce0?}, 0xffffffffffffb29c)
    pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go:2549 +0x158
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2.(*replicaSendStream).handleReadyEntriesRaftMuAndStreamLocked(0x4010b84000, {0x83ede40, 0x400b6cfce0}, {0x0, {0x1, 0x8d, 0x8e}, 0x94, {0x4002c6a000, 0x0, ...}, ...}, ...)
    pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go:2631 +0x808
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2.(*replicaState).handleReadyEntriesRaftMuLocked(0x4004d21440, {0x83ede40, 0x400b6cfce0}, {0x0, {0x1, 0x8d, 0x8e}, 0x94, {0x4002c6a000, 0x0, ...}, ...}, ...)
    pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go:2333 +0x134
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/rac2.(*rangeController).HandleRaftEventRaftMuLocked(0x4012ea2000, {0x83ede40, 0x400b6cfce0}, {0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, 0x400b6cfdd0, ...})
    pkg/kv/kvserver/kvflowcontrol/rac2/range_controller.go:1183 +0x8a4
github.com/cockroachdb/cockroach/pkg/kv/kvserver/kvflowcontrol/replica_rac2.(*processorImpl).HandleRaftReadyRaftMuLocked(0x4013cb7208, {0x83ede40, 0x400b6cfce0}, {0x6, 0x1, 0x2, 0x94, 0x2}, {0x0, 0x0, ...})
    pkg/kv/kvserver/kvflowcontrol/replica_rac2/processor.go:784 +0x178
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReadyRaftMuLocked(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    pkg/kv/kvserver/replica_raft.go:1017 +0x4b4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Replica).handleRaftReady(_, {_, _}, {{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...}, ...})
    pkg/kv/kvserver/replica_raft.go:836 +0x140
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*Store).processReady(0x40125f0408, 0x83ede40?)
    pkg/kv/kvserver/store_raft.go:682 +0xd4
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftSchedulerShard).worker(0x4006c2a3f0, {0x83ede40, 0x400e5b2de0}, {0x841c120, 0x40125f0408}, 0x4009f12008)
    pkg/kv/kvserver/scheduler.go:419 +0x240
github.com/cockroachdb/cockroach/pkg/kv/kvserver.(*raftScheduler).Start.func2({0x83ede40?, 0x400e5b2de0?})
    pkg/kv/kvserver/scheduler.go:319 +0x4c
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2({0x6a26144?, 0x6161616161616161?})
    pkg/util/stop/stopper.go:498 +0x198
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx in goroutine 58769
    pkg/util/stop/stopper.go:488 +0x384
Help

See also: [How To Investigate a Go Test Failure \(internal\)](https://cockroachlabs.atlassian.net/l/c/HgfXfJgM)

This test on roachdash | Improve this report!

Jira issue: CRDB-44467

yuzefovich commented 2 hours ago

dup of #135289