kv/kvserver/rangefeed: TestUnbufferedSenderDisconnect failed

cockroach-teamcity commented 10 hours ago

kv/kvserver/rangefeed.TestUnbufferedSenderDisconnect failed on master @ feec8b9a9cf1fdb35b60bda3bba5733e6a9e434a:

Previous read at 0x00c000243950 by goroutine 676:
  runtime.mapiterinit()
      GOROOT/src/runtime/map.go:816 +0x0
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*testServerStream).String()
      pkg/kv/kvserver/rangefeed/sender_helper_test.go:107 +0xc8
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.TestUnbufferedSenderDisconnect.func1.(*testServerStream).waitForEvent.2()
      pkg/kv/kvserver/rangefeed/sender_helper_test.go:84 +0x64
  github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1()
      pkg/testutils/soon.go:71 +0x57
  github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration()
      pkg/util/retry/retry.go:213 +0x175
  github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError()
      pkg/testutils/soon.go:77 +0xfe
  github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin()
      pkg/testutils/soon.go:56 +0x71
  github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon()
      pkg/testutils/soon.go:38 +0x54
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*testServerStream).waitForEvent()
      pkg/kv/kvserver/rangefeed/sender_helper_test.go:80 +0x4a4
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.TestUnbufferedSenderDisconnect.func1()
      pkg/kv/kvserver/rangefeed/unbuffered_sender_test.go:77 +0x41b
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Goroutine 675 (running) created at:
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx()
      pkg/util/stop/stopper.go:488 +0x72f
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask()
      pkg/util/stop/stopper.go:359 +0x248
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*StreamManager).Start()
      pkg/kv/kvserver/rangefeed/stream_manager.go:165 +0x13f
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.TestUnbufferedSenderDisconnect()
      pkg/kv/kvserver/rangefeed/unbuffered_sender_test.go:53 +0x591
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Goroutine 676 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1742 +0x825
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.TestUnbufferedSenderDisconnect()
      pkg/kv/kvserver/rangefeed/unbuffered_sender_test.go:55 +0x7c4
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44
==================

Parameters:

attempt=1
race=true
run=1
shard=1
Help

/cc @cockroachdb/kv _{This test on roachdash | Improve this report!

Jira issue: CRDB-44777}

kvoli commented 6 hours ago

Pattern matching here, but could this be related to https://github.com/cockroachdb/cockroach/commit/d65f62899938ff84709ee280d16e546e66e82915 @wenyihu6 / @stevendanna?

edit: I didn't actually read the failure, just git blame around the test. Looks like a data race so probably is something recent.

wenyihu6 commented 6 hours ago

Thanks for flagging. I came across a similar data race error and fixed it in my recent PR https://github.com/cockroachdb/cockroach/pull/135075. I'll take a closer look at this issue to determine if it's the same one.

wenyihu6 commented 5 hours ago

I was able to repro this data race locally and confirmed that the test passed across 1000 runs with the fix above applied. I will close this as being fixed. Feel free to send any other similar errors my way.

On feec8b9a9cf1fdb35b60bda3bba5733e6a9e434a,

==================== Test output for //pkg/kv/kvserver/rangefeed:rangefeed_test (run 30 of 1000):
==================
WARNING: DATA RACE
Write at 0x00c000709e60 by goroutine 46:
  runtime.mapassign_fast64()
      GOROOT/src/runtime/map_fast64.go:93 +0x0
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*testServerStream).Send()
      pkg/kv/kvserver/rangefeed/sender_helper_test.go:121 +0x15a
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*UnbufferedSender).run()
      pkg/kv/kvserver/rangefeed/unbuffered_sender.go:156 +0x247
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*StreamManager).Start.func1()
      pkg/kv/kvserver/rangefeed/stream_manager.go:167 +0x15c
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2()
      pkg/util/stop/stopper.go:498 +0x338
  github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.gowrap1()
      pkg/util/stop/stopper.go:499 +0x4f

Previous read at 0x00c000709e60 by goroutine 47:
  runtime.mapiterinit()
      GOROOT/src/runtime/map.go:816 +0x0
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.(*testServerStream).String()
      pkg/kv/kvserver/rangefeed/sender_helper_test.go:107 +0xc8
  github.com/cockroachdb/cockroach/pkg/kv/kvserver/rangefeed.TestUnbufferedSenderDisconnect.func1.(*testServerStream).waitForEvent.2()
      pkg/kv/kvserver/rangefeed/sender_helper_test.go:84 +0x64
  github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError.func1()
      pkg/testutils/soon.go:71 +0x57
  github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration()
      pkg/util/retry/retry.go:213 +0x175
  github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithinError()
      pkg/testutils/soon.go:77 +0xfe
  github.com/cockroachdb/cockroach/pkg/testutils.SucceedsWithin()
      pkg/testutils/soon.go:56 +0x71
  github.com/cockroachdb/cockroach/pkg/testutils.SucceedsSoon()

With the fix,

INFO: Build completed successfully, 1001 total actions
//pkg/kv/kvserver/rangefeed:rangefeed_test                               PASSED in 22.6s
  Stats over 1000 runs: max = 22.6s, min = 12.3s, avg = 19.0s, dev = 1.8s

cockroachdb / cockroach

kv/kvserver/rangefeed: TestUnbufferedSenderDisconnect failed #135895