search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.06k stars 3.8k forks source link

kv/kvclient/rangefeed: TestWithOnFrontierAdvance failed #117611

Closed cockroach-teamcity closed 9 months ago

cockroach-teamcity commented 9 months ago

kv/kvclient/rangefeed.TestWithOnFrontierAdvance failed with artifacts on master @ dab086d9d1d3355ad381ec493c73cabfa8d6b046:

      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:205 +0x2f7
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x278

Goroutine 13708 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x638
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xee4
  github.com/cockroachdb/cockroach/pkg/server.(*channelOrchestrator).startControlledServer()
      github.com/cockroachdb/cockroach/pkg/server/server_controller_channel_orchestrator.go:296 +0x9e9
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).createServerEntryLocked()
      github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:173 +0x2b2
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).startMissingServers()
      github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:134 +0x2f1
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).start.func1()
      github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:64 +0x1c9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x278

Goroutine 11345 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x638
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0x1ca
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:198 +0x4f
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).Start()
      github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:263 +0xf18
  github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
      github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1454 +0x339
  github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:2052 +0x5785
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:806 +0x104
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).PreStart()
      github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:20 +0x73
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:417 +0x57e
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:238 +0xa4
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:237 +0x84
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed_test.TestWithOnFrontierAdvance()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed_test/pkg/kv/kvclient/rangefeed/rangefeed_external_test.go:163 +0x175
  testing.tRunner()
      GOROOT/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1648 +0x44
==================

Parameters:

See also: How To Investigate a Go Test Failure (internal)

/cc @cockroachdb/replication

This test on roachdash | Improve this report!

Jira issue: CRDB-35270

cockroach-teamcity commented 9 months ago

kv/kvclient/rangefeed.TestWithOnFrontierAdvance failed with artifacts on master @ ee3495831f15ed85eeb8604c96e48b4f112b4f22:

      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:205 +0x2f7
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x278

Goroutine 12264 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x638
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0xee4
  github.com/cockroachdb/cockroach/pkg/server.(*channelOrchestrator).startControlledServer()
      github.com/cockroachdb/cockroach/pkg/server/server_controller_channel_orchestrator.go:296 +0x9e9
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).createServerEntryLocked()
      github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:173 +0x2b2
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).startMissingServers()
      github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:134 +0x2f1
  github.com/cockroachdb/cockroach/pkg/server.(*serverController).start.func1()
      github.com/cockroachdb/cockroach/pkg/server/server_controller_orchestration.go:64 +0x1c9
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:484 +0x278

Goroutine 10125 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:475 +0x638
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:346 +0x1ca
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache.Start()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed/rangefeedcache/watcher.go:198 +0x4f
  github.com/cockroachdb/cockroach/pkg/server/settingswatcher.(*SettingsWatcher).Start()
      github.com/cockroachdb/cockroach/pkg/server/settingswatcher/settings_watcher.go:263 +0xf18
  github.com/cockroachdb/cockroach/pkg/server.(*SQLServer).preStart()
      github.com/cockroachdb/cockroach/pkg/server/server_sql.go:1454 +0x339
  github.com/cockroachdb/cockroach/pkg/server.(*topLevelServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/server.go:2052 +0x5785
  github.com/cockroachdb/cockroach/pkg/server.(*testServer).PreStart()
      github.com/cockroachdb/cockroach/pkg/server/testserver.go:807 +0x104
  github.com/cockroachdb/cockroach/pkg/testutils/serverutils.(*wrap).PreStart()
      github.com/cockroachdb/cockroach/bazel-out/k8-fastbuild/bin/pkg/testutils/serverutils/ts_control_forwarder_generated.go:20 +0x73
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.(*TestCluster).Start()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:417 +0x57e
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:238 +0xa4
  github.com/cockroachdb/cockroach/pkg/testutils/testcluster.StartTestCluster()
      github.com/cockroachdb/cockroach/pkg/testutils/testcluster/testcluster.go:237 +0x84
  github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed_test.TestWithOnFrontierAdvance()
      github.com/cockroachdb/cockroach/pkg/kv/kvclient/rangefeed_test/pkg/kv/kvclient/rangefeed/rangefeed_external_test.go:163 +0x175
  testing.tRunner()
      GOROOT/src/testing/testing.go:1595 +0x261
  testing.(*T).Run.func1()
      GOROOT/src/testing/testing.go:1648 +0x44
==================

Parameters:

See also: How To Investigate a Go Test Failure (internal)

This test on roachdash | Improve this report!

miraradeva commented 9 months ago

Adding label P1 to match the original instance of this data race in https://github.com/cockroachdb/cockroach/issues/117688.