search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.87k stars 3.77k forks source link

roachtest: cdc/crdb-chaos failed #77815

Closed cockroach-teamcity closed 1 year ago

cockroach-teamcity commented 2 years ago

roachtest.cdc/crdb-chaos failed with artifacts on master @ f5fc84fb5707428ae9505c5e3e90cf3f63d465ad:

          |  1710.0s     6601            2.0            2.0     46.1     46.1     46.1     46.1 delivery
          |  1710.0s     6601           22.0           20.3     19.9     25.2     25.2     25.2 newOrder
          |  1710.0s     6601            0.0            2.0      0.0      0.0      0.0      0.0 orderStatus
          |  1710.0s     6601           19.0           19.7     12.6     14.2     14.7     14.7 payment
          |  1710.0s     6601            3.0            2.0      7.6      8.9      8.9      8.9 stockLevel
        Wraps: (4) secondary error attachment
          | UNCLASSIFIED_PROBLEM: context canceled
          | (1) UNCLASSIFIED_PROBLEM
          | Wraps: (2) Node 4. Command with error:
          |   | ``````
          |   | ./workload run tpcc --warehouses=100 --duration=30m --tolerate-errors {pgurl:1-3}
          |   | ``````
          | Wraps: (3) context canceled
          | Error types: (1) errors.Unclassified (2) *hintdetail.withDetail (3) *errors.errorString
        Wraps: (5) context canceled
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *cluster.WithCommandDetails (4) *secondary.withSecondaryError (5) *errors.errorString

    monitor.go:127,cdc.go:296,cdc.go:759,test_runner.go:866: monitor failure: monitor task failed: unexpected status: failed
        (1) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).WaitE
          |     main/pkg/cmd/roachtest/monitor.go:115
          | main.(*monitorImpl).Wait
          |     main/pkg/cmd/roachtest/monitor.go:123
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:296
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func5
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:759
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:866
        Wraps: (2) monitor failure
        Wraps: (3) attached stack trace
          -- stack trace:
          | main.(*monitorImpl).wait.func2
          |     main/pkg/cmd/roachtest/monitor.go:171
        Wraps: (4) monitor task failed
        Wraps: (5) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*latencyVerifier).pollLatency
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:1699
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest.func3
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:274
          | main.(*monitorImpl).Go.func1
          |     main/pkg/cmd/roachtest/monitor.go:105
          | golang.org/x/sync/errgroup.(*Group).Go.func1
          |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:57
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (6) unexpected status: failed
        Error types: (1) *withstack.withStack (2) *errutil.withPrefix (3) *withstack.withStack (4) *errutil.withPrefix (5) *withstack.withStack (6) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #68047 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-21.1]

/cc @cockroachdb/cdc

This test on roachdash | Improve this report!

Jira issue: CRDB-13780

Epic CRDB-11732

miretskiy commented 2 years ago

No good deed goes unpunished; you recently worked w/ code around target resolution; seems to be failing now.

cockroach-teamcity commented 2 years ago

roachtest.cdc/crdb-chaos failed with artifacts on master @ 29716850b181718594663889ddb5f479fef7a305:

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /artifacts/cdc/crdb-chaos/run_1
    cluster.go:1868,cdc.go:114,cdc.go:759,test_runner.go:875: one or more parallel execution failure
        (1) attached stack trace
          -- stack trace:
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).ParallelE
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:2042
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Parallel
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cluster_synced.go:1923
          | github.com/cockroachdb/cockroach/pkg/roachprod/install.(*SyncedCluster).Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/install/cockroach.go:167
          | github.com/cockroachdb/cockroach/pkg/roachprod.Start
          |     github.com/cockroachdb/cockroach/pkg/roachprod/roachprod.go:660
          | main.(*clusterImpl).StartE
          |     main/pkg/cmd/roachtest/cluster.go:1826
          | main.(*clusterImpl).Start
          |     main/pkg/cmd/roachtest/cluster.go:1867
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.cdcBasicTest
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:114
          | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerCDC.func5
          |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cdc.go:759
          | main.(*testRunner).runTest.func2
          |     main/pkg/cmd/roachtest/test_runner.go:875
          | runtime.goexit
          |     GOROOT/src/runtime/asm_amd64.s:1581
        Wraps: (2) one or more parallel execution failure
        Error types: (1) *withstack.withStack (2) *errutil.leafError
Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #68047 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cdc/crdb-chaos failed with artifacts on master @ a80652b2e4691ea76ea49e797b1b9e0998e1d61f:

test artifacts and logs in: /artifacts/cdc/crdb-chaos/run_1
(test_impl.go:297).Fatalf: max latency was more than allowed: 5m47.19389053s vs 5m0s
(test_impl.go:291).Fatal: monitor failure: monitor task failed: t.Fatal() was called

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #68047 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cdc/crdb-chaos failed with artifacts on master @ 0725273ac7f789ba8ed78aacaf73cc953ca47fe8:

test artifacts and logs in: /artifacts/cdc/crdb-chaos/run_1
(cdc.go:373).newChangefeed: failed to create changefeed: pq: Use of CHANGEFEED requires an enterprise license. Your evaluation license expired on December 30, 2022. If you're interested in getting a new license, please contact subscriptions@cockroachlabs.com and we can help you out.
(cluster.go:1933).Run: output in run_075002.193383933_n4_workload_run_tpcc: ./workload run tpcc --warehouses=100 --duration=30m --tolerate-errors {pgurl:1-3}  returned: context canceled
(cdc.go:283).Close: error shutting down prometheus/grafana: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #94489 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.2 release-blocker] - #94476 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.1 release-blocker] - #68047 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cdc/crdb-chaos failed with artifacts on master @ 0725273ac7f789ba8ed78aacaf73cc953ca47fe8:

test artifacts and logs in: /artifacts/cdc/crdb-chaos/run_1
(cdc.go:373).newChangefeed: failed to create changefeed: pq: Use of CHANGEFEED requires an enterprise license. Your evaluation license expired on December 30, 2022. If you're interested in getting a new license, please contact subscriptions@cockroachlabs.com and we can help you out.
(cluster.go:1933).Run: output in run_074852.872265925_n4_workload_run_tpcc: ./workload run tpcc --warehouses=100 --duration=30m --tolerate-errors {pgurl:1-3}  returned: context canceled
(cdc.go:283).Close: error shutting down prometheus/grafana: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #94489 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.2 release-blocker] - #94476 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.1 release-blocker] - #68047 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cdc/crdb-chaos failed with artifacts on master @ 0725273ac7f789ba8ed78aacaf73cc953ca47fe8:

test artifacts and logs in: /artifacts/cdc/crdb-chaos/run_1
(cdc.go:373).newChangefeed: failed to create changefeed: pq: Use of CHANGEFEED requires an enterprise license. Your evaluation license expired on December 30, 2022. If you're interested in getting a new license, please contact subscriptions@cockroachlabs.com and we can help you out.
(cluster.go:1933).Run: output in run_080127.645193117_n4_workload_run_tpcc: ./workload run tpcc --warehouses=100 --duration=30m --tolerate-errors {pgurl:1-3}  returned: context canceled
(cdc.go:283).Close: error shutting down prometheus/grafana: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #94489 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.2 release-blocker] - #94476 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.1 release-blocker] - #68047 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-21.1]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 year ago

roachtest.cdc/crdb-chaos failed with artifacts on master @ 1d7bd69205c2197ccac33df9e2e6d4ff8c0fdbcf:

test artifacts and logs in: /artifacts/cdc/crdb-chaos/run_1
(cdc.go:373).newChangefeed: failed to create changefeed: pq: Use of CHANGEFEED requires an enterprise license. Your evaluation license expired on December 30, 2022. If you're interested in getting a new license, please contact subscriptions@cockroachlabs.com and we can help you out.
(cluster.go:1933).Run: output in run_075924.988588676_n4_workload_run_tpcc: ./workload run tpcc --warehouses=100 --duration=30m --tolerate-errors {pgurl:1-3}  returned: context canceled
(cdc.go:283).Close: error shutting down prometheus/grafana: context canceled

Parameters: ROACHTEST_cloud=gce , ROACHTEST_cpu=16 , ROACHTEST_encrypted=false , ROACHTEST_fs=ext4 , ROACHTEST_localSSD=true , ROACHTEST_ssd=0

Help

See: [roachtest README](https://github.com/cockroachdb/cockroach/blob/master/pkg/cmd/roachtest/README.md) See: [How To Investigate \(internal\)](https://cockroachlabs.atlassian.net/l/c/SSSBr8c7)

Same failure on other branches

- #94489 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.2 release-blocker] - #94476 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-22.1 release-blocker] - #68047 roachtest: cdc/crdb-chaos failed [C-test-failure O-roachtest O-robot T-cdc branch-release-21.1]

This test on roachdash | Improve this report!

miretskiy commented 1 year ago

Closing as stale.