ccl/changefeedccl: TestParallelIOMetrics failed

cockroach-teamcity commented 1 month ago

ccl/changefeedccl.TestParallelIOMetrics failed on release-24.2 @ ebfc78b8e14f0d4c3df0b8505fb766c5a434f08e:

=== RUN   TestParallelIOMetrics
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestParallelIOMetrics1852613363
    test_log_scope.go:81: use -show-logs to present logs inline
    changefeed_test.go:9299: -- test log scope end --
test logs left over in: outputs.zip/logTestParallelIOMetrics1852613363
--- FAIL: TestParallelIOMetrics (163.31s)
=== RUN   TestParallelIOMetrics/pubsub
    helpers_test.go:872: making server as secondary tenant
    helpers_test.go:951: making pubsub feed factory
    changefeed_test.go:9295: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:9295
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1123
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1158
            Error:          Received unexpected error:
                            context deadline exceeded
            Test:           TestParallelIOMetrics/pubsub
    --- FAIL: TestParallelIOMetrics/pubsub (163.30s)

Parameters:

attempt=1
deadlock=true
run=1
shard=5
Help

Same failure on other branches

- #125295 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot P-3 T-cdc branch-master]

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-41769

cockroach-teamcity commented 1 month ago

ccl/changefeedccl.TestParallelIOMetrics failed on release-24.2 @ 3542ba5a3f47f9042e81583fe52c893aac9c4e3c:

=== RUN   TestParallelIOMetrics
    test_log_scope.go:170: test logs captured to: outputs.zip/logTestParallelIOMetrics649852617
    test_log_scope.go:81: use -show-logs to present logs inline
    changefeed_test.go:9300: -- test log scope end --
test logs left over in: outputs.zip/logTestParallelIOMetrics649852617
--- FAIL: TestParallelIOMetrics (161.70s)
=== RUN   TestParallelIOMetrics/pubsub
    helpers_test.go:872: making server as secondary tenant
    helpers_test.go:951: making pubsub feed factory
    changefeed_test.go:9296: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:9296
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1123
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1158
            Error:          Received unexpected error:
                            context deadline exceeded
            Test:           TestParallelIOMetrics/pubsub
    --- FAIL: TestParallelIOMetrics/pubsub (161.69s)

Parameters:

attempt=1
deadlock=true
run=1
shard=5
Help

Same failure on other branches

- #130390 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot T-cdc branch-release-24.2.2-rc release-blocker] - #125295 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

ccl/changefeedccl.TestParallelIOMetrics failed on release-24.2 @ 08468de10d89483d52fecfcf3fb3892564e8df3a:

=== RUN   TestParallelIOMetrics
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestParallelIOMetrics750364326
    test_log_scope.go:76: use -show-logs to present logs inline
    changefeed_test.go:9381: -- test log scope end --
test logs left over in: outputs.zip/logTestParallelIOMetrics750364326
--- FAIL: TestParallelIOMetrics (243.13s)
=== RUN   TestParallelIOMetrics/pubsub
    helpers_test.go:869: making server as secondary tenant
    helpers_test.go:948: making pubsub feed factory
    changefeed_test.go:9353: condition failed to evaluate within 3m45s: from changefeed_test.go:9356: waiting for pending keys: 0
    --- FAIL: TestParallelIOMetrics/pubsub (243.12s)

Parameters:

attempt=1
deadlock=true
run=3
shard=6
Help

Same failure on other branches

- #130390 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot T-cdc branch-release-24.2.2-rc release-blocker] - #125295 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

ccl/changefeedccl.TestParallelIOMetrics failed on release-24.2 @ 30993f0fc73fa753d53711077a2bfd9fe3f0255e:

=== RUN   TestParallelIOMetrics
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestParallelIOMetrics1178914659
    test_log_scope.go:76: use -show-logs to present logs inline
    changefeed_test.go:9381: -- test log scope end --
test logs left over in: outputs.zip/logTestParallelIOMetrics1178914659
--- FAIL: TestParallelIOMetrics (192.78s)
=== RUN   TestParallelIOMetrics/pubsub
    helpers_test.go:869: making server as secondary tenant
    helpers_test.go:948: making pubsub feed factory
    changefeed_test.go:9377: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:9377
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1120
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1155
            Error:          Received unexpected error:
                            context deadline exceeded
            Test:           TestParallelIOMetrics/pubsub
    --- FAIL: TestParallelIOMetrics/pubsub (192.78s)

Parameters:

attempt=1
deadlock=true
run=2
shard=6
Help

Same failure on other branches

- #130390 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot T-cdc branch-release-24.2.2-rc release-blocker] - #125295 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

ccl/changefeedccl.TestParallelIOMetrics failed on release-24.2 @ c3712a42b57bcd6dd4285fc52d915fd4dc931e6e:

=== RUN   TestParallelIOMetrics
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestParallelIOMetrics732720319
    test_log_scope.go:76: use -show-logs to present logs inline
    changefeed_test.go:9381: -- test log scope end --
test logs left over in: outputs.zip/logTestParallelIOMetrics732720319
--- FAIL: TestParallelIOMetrics (172.25s)
=== RUN   TestParallelIOMetrics/pubsub
    helpers_test.go:869: making server as secondary tenant
    helpers_test.go:948: making pubsub feed factory
    changefeed_test.go:9377: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:9377
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1120
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1155
            Error:          Received unexpected error:
                            context deadline exceeded
            Test:           TestParallelIOMetrics/pubsub
    --- FAIL: TestParallelIOMetrics/pubsub (172.25s)

Parameters:

attempt=1
deadlock=true
run=1
shard=6
Help

Same failure on other branches

- #130390 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot T-cdc branch-release-24.2.2-rc release-blocker] - #125295 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

ccl/changefeedccl.TestParallelIOMetrics failed on release-24.2 @ 5405e9098bbc9e12ccc4387664eaedfc2d32e9d9:

=== RUN   TestParallelIOMetrics
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestParallelIOMetrics144548274
    test_log_scope.go:76: use -show-logs to present logs inline
    changefeed_test.go:9381: -- test log scope end --
test logs left over in: outputs.zip/logTestParallelIOMetrics144548274
--- FAIL: TestParallelIOMetrics (193.15s)
=== RUN   TestParallelIOMetrics/pubsub
    helpers_test.go:869: making server as secondary tenant
    helpers_test.go:948: making pubsub feed factory
    changefeed_test.go:9377: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:9377
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1120
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1155
            Error:          Received unexpected error:
                            context deadline exceeded
            Test:           TestParallelIOMetrics/pubsub
    --- FAIL: TestParallelIOMetrics/pubsub (193.13s)

Parameters:

attempt=1
deadlock=true
run=2
shard=6
Help

Same failure on other branches

- #130390 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot T-cdc branch-release-24.2.2-rc release-blocker] - #125295 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 week ago

ccl/changefeedccl.TestParallelIOMetrics failed on release-24.2 @ 52fafd7ae8a078124f983d0c60550e9f38c68462:

=== RUN   TestParallelIOMetrics
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestParallelIOMetrics2066210130
    test_log_scope.go:76: use -show-logs to present logs inline
    changefeed_test.go:9381: -- test log scope end --
test logs left over in: outputs.zip/logTestParallelIOMetrics2066210130
--- FAIL: TestParallelIOMetrics (169.60s)
=== RUN   TestParallelIOMetrics/pubsub
    helpers_test.go:869: making server as secondary tenant
    helpers_test.go:948: making pubsub feed factory
    changefeed_test.go:9377: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:9377
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1120
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1155
            Error:          Received unexpected error:
                            context deadline exceeded
            Test:           TestParallelIOMetrics/pubsub
    --- FAIL: TestParallelIOMetrics/pubsub (169.60s)

Parameters:

attempt=1
deadlock=true
run=1
shard=6
Help

Same failure on other branches

- #130390 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot T-cdc branch-release-24.2.2-rc release-blocker] - #125295 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 days ago

ccl/changefeedccl.TestParallelIOMetrics failed on release-24.2 @ d631a3bd06b4068b40e010ea55ee7813abaa8fe4:

=== RUN   TestParallelIOMetrics
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestParallelIOMetrics1174321141
    test_log_scope.go:76: use -show-logs to present logs inline
    changefeed_test.go:9370: -- test log scope end --
test logs left over in: outputs.zip/logTestParallelIOMetrics1174321141
--- FAIL: TestParallelIOMetrics (158.82s)
=== RUN   TestParallelIOMetrics/pubsub
    helpers_test.go:869: making server as secondary tenant
    helpers_test.go:948: making pubsub feed factory
    changefeed_test.go:9366: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:9366
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1120
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1155
            Error:          Received unexpected error:
                            context deadline exceeded
            Test:           TestParallelIOMetrics/pubsub
    --- FAIL: TestParallelIOMetrics/pubsub (158.81s)

Parameters:

attempt=1
deadlock=true
run=1
shard=6
Help

Same failure on other branches

- #130390 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot P-2 T-cdc X-duplicate branch-release-24.2.2-rc release-blocker] - #125295 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 days ago

ccl/changefeedccl.TestParallelIOMetrics failed on release-24.2 @ d631a3bd06b4068b40e010ea55ee7813abaa8fe4:

=== RUN   TestParallelIOMetrics
    test_log_scope.go:165: test logs captured to: outputs.zip/logTestParallelIOMetrics1309902973
    test_log_scope.go:76: use -show-logs to present logs inline
    changefeed_test.go:9370: -- test log scope end --
test logs left over in: outputs.zip/logTestParallelIOMetrics1309902973
--- FAIL: TestParallelIOMetrics (222.05s)
=== RUN   TestParallelIOMetrics/pubsub
    helpers_test.go:869: making server as secondary tenant
    helpers_test.go:948: making pubsub feed factory
    changefeed_test.go:9366: 
            Error Trace:    github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:9366
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1120
                                        github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1155
            Error:          Received unexpected error:
                            context deadline exceeded
            Test:           TestParallelIOMetrics/pubsub
    --- FAIL: TestParallelIOMetrics/pubsub (222.05s)

Parameters:

attempt=1
deadlock=true
run=2
shard=6
Help

Same failure on other branches

- #130390 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot P-2 T-cdc X-duplicate branch-release-24.2.2-rc release-blocker] - #125295 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 days ago

ccl/changefeedccl.TestParallelIOMetrics failed on release-24.2 @ d631a3bd06b4068b40e010ea55ee7813abaa8fe4:

      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:9366 +0x679
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamed.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1120 +0x93
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamedWithSystem.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1155 +0x46d
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Previous read at 0x00001447dfa8 by goroutine 119557:
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:248 +0x64
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.func2()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:274 +0xf3
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO.Group.GoCtx.func6()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:163 +0x46
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x91

Goroutine 112237 (running) created at:
  testing.(*T).Run()
      GOROOT/src/testing/testing.go:1742 +0x825
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamedWithSystem()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1145 +0x4f0
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTestNamed()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1122 +0xbd
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.cdcTest()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/helpers_test.go:1115 +0x18e
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.TestParallelIOMetrics()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/changefeed_test.go:9369 +0x111
  testing.tRunner()
      GOROOT/src/testing/testing.go:1689 +0x21e
  testing.(*T).Run.gowrap1()
      GOROOT/src/testing/testing.go:1742 +0x44

Goroutine 119557 (running) created at:
  golang.org/x/sync/errgroup.(*Group).Go()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75 +0x124
  github.com/cockroachdb/cockroach/pkg/util/ctxgroup.Group.GoCtx()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:162 +0x1be
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.(*ParallelIO).processIO()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:272 +0x1dd
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.NewParallelIO.func1()
      github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl/parallel_io.go:101 +0x4c
  github.com/cockroachdb/cockroach/pkg/ccl/changefeedccl.NewParallelIO.Group.GoCtx.func2()
      github.com/cockroachdb/cockroach/pkg/util/ctxgroup/ctxgroup.go:163 +0x46
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78 +0x91
==================

Parameters:

attempt=1
race=true
run=3
shard=6
Help

Same failure on other branches

- #130390 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot P-2 T-cdc X-duplicate branch-release-24.2.2-rc release-blocker] - #125295 ccl/changefeedccl: TestParallelIOMetrics failed [A-cdc C-test-failure O-robot P-3 T-cdc branch-master]

This test on roachdash | Improve this report!

cockroachdb / cockroach

ccl/changefeedccl: TestParallelIOMetrics failed #129917