Closed cockroach-teamcity closed 1 month ago
Failed on a data race in storage:
WARNING: DATA RACE
Write at 0x00c0041b2db0 by goroutine 20:
github.com/cockroachdb/cockroach/pkg/storage/disk.(*Monitor).Close()
github.com/cockroachdb/cockroach/pkg/storage/disk/monitor.go:292 +0x8e
github.com/cockroachdb/cockroach/pkg/server.(*diskStatsMap).closeDiskMonitors()
github.com/cockroachdb/cockroach/pkg/server/node.go:1126 +0x113
github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart.func9()
github.com/cockroachdb/cockroach/pkg/server/server.go:1973 +0x1a
github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:108 +0x26
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:550 +0x259
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).stopServerLocked()
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:226 +0xe4
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).stopServers()
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:163 +0x42d
github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start.func3()
github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:453 +0x3c
github.com/cockroachdb/cockroach/pkg/util/stop.CloserFn.Close()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:108 +0x26
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).Stop()
github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:550 +0x259
github.com/cockroachdb/cockroach/pkg/cli.TestCollectInfoFromMultipleStores()
github.com/cockroachdb/cockroach/pkg/cli/debug_recover_loss_of_quorum_test.go:82 +0x7f2
testing.tRunner()
GOROOT/src/testing/testing.go:1689 +0x21e
testing.(*T).Run.gowrap1()
GOROOT/src/testing/testing.go:1742 +0x44
Previous read at 0x00c0041b2db0 by goroutine 3356:
github.com/cockroachdb/cockroach/pkg/storage/disk.(*Monitor).CumulativeStats()
github.com/cockroachdb/cockroach/pkg/storage/disk/monitor.go:237 +0x84
github.com/cockroachdb/cockroach/pkg/server.(*diskStatsMap).tryPopulateAdmissionDiskStats()
github.com/cockroachdb/cockroach/pkg/server/node.go:1071 +0x2b9
github.com/cockroachdb/cockroach/pkg/server.(*Node).GetPebbleMetrics()
github.com/cockroachdb/cockroach/pkg/server/node.go:1153 +0xd1
github.com/cockroachdb/cockroach/pkg/util/admission.(*StoreGrantCoordinators).SetPebbleMetricsProvider.func1()
github.com/cockroachdb/cockroach/pkg/util/admission/grant_coordinator.go:118 +0x177
Re-assigning to @cockroachdb/storage
This is a race on shutdown, where the stopper may close the disk.Monitor before admission control is done reading it (via Node.GetPebbleMetrics
). Removing the release-blocker label.
I'm going to look at refactoring this so that admission control owns its own disk.Monitor (which is a lightweight type that will result in a separate ref of the underlying monitoredDisk
).
cli.TestCollectInfoFromMultipleStores failed on release-24.2 @ 191522a802ca853ed6776fc18d8c4d8f38599bbb:
Parameters:
attempt=1
race=true
run=1
shard=10
Help
See also: How To Investigate a Go Test Failure (internal)
/cc @cockroachdb/kv @cockroachdb/serverThis test on roachdash | Improve this report!
Jira issue: CRDB-41303