search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.9k stars 3.78k forks source link

roachtest: c2c/BulkOps/full failed #116747

Open cockroach-teamcity opened 9 months ago

cockroach-teamcity commented 9 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 5f76d93b2ad8d29c14c9b176753ec33cbf81e3fe:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:751
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:972
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1263
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:747
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:729
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:972
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1263
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-34750

cockroach-teamcity commented 9 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ d2d974202781321ddb4fb1115c68fe7566ab9d06:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:751
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:972
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1263
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:747
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:729
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:972
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1263
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ f2917d82b235db6fda145a932050e5a58f957b9f:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:751
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:972
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1263
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:747
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:729
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:972
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1263
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 677cd7db760cc1a0793064da7c2e6c03b14e33a6:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:751
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:972
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1263
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:747
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:729
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:972
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1263
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

stevendanna commented 8 months ago

I've marked this as P3 since improving our import performance is currently not planned work for the near future.

cockroach-teamcity commented 8 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ d43afb2dbde0658d1dabb7b576f0e5afe149a672:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:751
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:747
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:729
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ ee3495831f15ed85eeb8604c96e48b4f112b4f22:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:751
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:747
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:729
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 75f19bec66a542d58680e9ccb47d57598f48fe20:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:751
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:747
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:729
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ c24aedd67a89ea14e91b085d37103127d693aa1a:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:751
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:747
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:729
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 3c837c2a86165188a26e8629b7e21b7ae0fb56ae:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:751
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:747
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:729
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 8 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 585ffa0acfa6dd771aad8320d3a9e219acf56656:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:751
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:747
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:729
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ e0fbbe4c1334580bf05d6d6fb0d406a327e43944:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:751
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:747
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:729
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ d7d442e4a3c9dca7e01c4c6f4f00e2f28faa4374:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:751
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:747
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:729
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 353fded9fe270b3eee4c85480ac1b9ec819f23b0:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:751
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:747
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:729
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:979
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1270
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 814a375d4c0e79d875c42452725f05f6c27294e3:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:752
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:748
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:730
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 254dbd247fb8ed352a11439063b29f23a0767f28:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:752
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:748
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:730
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 0b7ae19e2b94b851ed8812914f57032aab699811:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:752
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:748
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:730
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 7 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 04f0416d526a43741d22fd03966758dcccdeb79f:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:752
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:748
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:730
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ c9c3cc5f3c3a4a6ab556f4b9d5b6ec0381901bdb:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:752
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:748
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:730
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 7bb52a7d1c75d5adfdfa53e5fcff6f5e6497408f:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:752
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                                main/pkg/cmd/roachtest/monitor.go:119
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                                src/runtime/asm_amd64.s:1650
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:748
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:730
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:119
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:75
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1650
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:153).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 41da9eccc216413c09cc66ad4c6bfa5f92c506e3:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:752
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:748
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:730
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 2ac5657aef0fd878973191079b6f3aadbc9c9266:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:752
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:748
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:730
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 6 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ ebf92d22a7e4afe2423e681d49c615104b5f0098:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:752
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:748
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:730
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ c994982a8be5af89f594e115e897dd6d62cf99d8:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:752
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:748
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:730
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 5 months ago

roachtest.c2c/BulkOps/full failed with artifacts on master @ 7fc4c7bcbbf0c75a62d056da0bf79a5a32714650:

(assertions.go:333).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:752
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                                github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    expected job status succeeded, but got running
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream.func1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:748
                      | github.com/cockroachdb/cockroach/pkg/util/retry.ForDuration
                      |     github.com/cockroachdb/cockroach/pkg/util/retry/retry.go:213
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).stopReplicationStream
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:730
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.(*replicationDriver).main
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:980
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerClusterToCluster.func1.1
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/cluster_to_cluster.go:1271
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) expected job status succeeded, but got running
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           c2c/BulkOps/full
(require.go:1360).NoError: FailNow called
(monitor.go:154).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/c2c/BulkOps/full/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #121412 roachtest: c2c/BulkOps/full failed [C-test-failure O-roachtest O-robot T-disaster-recovery branch-release-24.1 release-blocker]

This test on roachdash | Improve this report!