roachtest: cdc/mixed-versions failed

cockroach-teamcity commented 1 month ago

roachtest.cdc/mixed-versions failed with artifacts on release-24.1 @ fbb20b1a7f21bf7a175d0717d7f5c76ee5722ce9:

(test_runner.go:1239).runTest: test timed out (1h0m0s)
test artifacts and logs in: /artifacts/cdc/mixed-versions/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #124742 roachtest: cdc/mixed-versions failed [A-cdc C-test-failure O-roachtest O-robot P-2 T-cdc branch-master]

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-40254

rharding6373 commented 1 month ago

  | (1) background step `run "run workload"` returned error
  | Wraps: (2) test failed:
  |   | test random seed: 5786693562605079591
  |   |
  |   |                                     n1          n2          n3          n4
  |   | released versions                   v23.1.7     v23.1.7     v23.1.7     v23.1.7
  |   | logical binary versions             24.1        23.2        24.1        24.1
  |   | cluster versions before failure     23.2        23.2        23.2        23.2

Timed out while running the test:

context canceled
(1) attached stack trace
  -- stack trace:
  | main.(*clusterImpl).Run
  |     main/pkg/cmd/roachtest/cluster.go:2398
  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.kafkaManager.stop
  |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:2649
  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.setupKafka.func1
  |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:3045
  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*cdcMixedVersionTester).StartKafka.func1
  |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_cdc.go:170
  | runtime.gopanic
  |     GOROOT/src/runtime/panic.go:770
  | main.(*testImpl).Fatal
  |     main/pkg/cmd/roachtest/test_impl.go:359
  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion.(*Test).Run
  |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/roachtestutil/mixedversion/mixedversion.go:620
  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.runCDCMixedVersions
  |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/[mixed_version_cdc.go:567](https://github.com/cockroachdb/cockroach/blob/fbb20b1a7f21bf7a175d0717d7f5c76ee5722ce9/pkg/cmd/roachtest/tests/mixed_version_cdc.go#L567)
  | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDCMixedVersions.func1
  |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/mixed_version_cdc.go:80
  | main.(*testRunner).runTest.func2
  |     main/pkg/cmd/roachtest/test_runner.go:1215
  | runtime.goexit
  |     src/runtime/asm_amd64.s:1695
Wraps: (2) secondary error attachment
  | context canceled
  | (1) context canceled
  | Error types: (1) *errors.errorString
Wraps: (3) context canceled
Error types: (1) *withstack.withStack (2) *secondary.withSecondaryError (3) *errutil.leafError

wenyihu6 commented 1 month ago

renatolabs commented 1 month ago

Fixed by #127096.

cockroachdb / cockroach

roachtest: cdc/mixed-versions failed #127051