search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
29.85k stars 3.77k forks source link

roachtest: kv/gracefuldraining failed #126986

Closed cockroach-teamcity closed 2 weeks ago

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 495d20e82f09b3436f1b00d7331146fe2042dd6a:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 48.00 at time 2024-07-11 09:22:21.441444532 +0000 UTC m=+12619.612331461 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 48.00 at time 2024-07-11 09:22:21.441444532 +0000 UTC m=+12619.612331461 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-40222

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ d6523e58ab3c09998535e88646ca16c1ead1ae4e:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 5.00 at time 2024-07-12 15:20:50.575458635 +0000 UTC m=+34158.397503346 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 5.00 at time 2024-07-12 15:20:50.575458635 +0000 UTC m=+34158.397503346 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

arulajmani commented 1 month ago

This is the same failure as . We see https://github.com/cockroachdb/cockroach/issues/125731.

Error: ERROR: result is ambiguous: error=failed to connect to n6 at 10.142.0.19:26257: grpc: error reading from server: EOF [code 14/Unavailable] [exhausted] (last error: failed to connect to n6 at 10.142.0.19:26257: grpc: error reading from server: EOF [code 14/Unavailable]) (SQLSTATE 40003)
run_092210.870628077_n7_cockroach-workload-r: 2024/07/11 09:22:20 cluster.go:2451: > result: COMMAND_PROBLEM: exit status 1
2024/07/11 09:22:19 cluster.go:692: test status: stopping nodes :6
teamcity-16022118-1720677126-35-n7cpu4: stopping
2024/07/11 09:22:20 kv.go:571: test worker status: workload command completed
2024/07/11 09:22:20 kv.go:637: test status: letting workload run with one node down
2024/07/11 09:22:21 test_impl.go:414: test failure #1: full stack retained in failure_1.log: (assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 48.00 at time 2024-07-11 09:22:21.441444532 +0000 UTC m=+12619.612331461 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 23edce4c9826ebd4711b71c1029d2fc38a25f974:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 455.00 at time 2024-07-13 08:42:27.768890966 +0000 UTC m=+10358.195053543 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 455.00 at time 2024-07-13 08:42:27.768890966 +0000 UTC m=+10358.195053543 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ d4ef0343f38a35fb781b6343697b57a58131c68e:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1159.00 at time 2024-07-14 09:07:13.865944372 +0000 UTC m=+11781.994145568 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1159.00 at time 2024-07-14 09:07:13.865944372 +0000 UTC m=+11781.994145568 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2354).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ d4ef0343f38a35fb781b6343697b57a58131c68e:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1605.00 at time 2024-07-15 08:10:44.93512217 +0000 UTC m=+8405.039687049 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1605.00 at time 2024-07-15 08:10:44.93512217 +0000 UTC m=+8405.039687049 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

andrewbaptist commented 1 month ago

Dupe of #125731 - however is happening fairly frequently. It likely needs furter investigation. Leaving as a P2. I'm going to assign myself and determine if this is related to proxy code.

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 238d10a22cff57ddc8393b149cffb58883a2a89f:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 579.00 at time 2024-07-16 08:09:28.802106011 +0000 UTC m=+8316.321745780 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 579.00 at time 2024-07-16 08:09:28.802106011 +0000 UTC m=+8316.321745780 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 037402f139a1d037d40cd9783431f3790fa3b75a:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 28.00 at time 2024-07-17 09:09:55.898572817 +0000 UTC m=+12012.188874939 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 28.00 at time 2024-07-17 09:09:55.898572817 +0000 UTC m=+12012.188874939 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 3a5a06709653819261ee998badf5efd6d3bd5955:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 593.00 at time 2024-07-19 09:34:44.682700742 +0000 UTC m=+13374.248277397 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 593.00 at time 2024-07-19 09:34:44.682700742 +0000 UTC m=+13374.248277397 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2354).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ f93f42b7cf5dead978cba81aece33c050bbe96a8:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 417.00 at time 2024-07-20 09:59:26.02661283 +0000 UTC m=+14952.315396496 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 417.00 at time 2024-07-20 09:59:26.02661283 +0000 UTC m=+14952.315396496 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 60d3fb84a73f5e31c99404223f3a0ba91e67f489:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1062.00 at time 2024-07-21 09:33:21.680267522 +0000 UTC m=+13393.952264953 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1062.00 at time 2024-07-21 09:33:21.680267522 +0000 UTC m=+13393.952264953 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 509b1e919fa3e89254b4b836854aaddc5c26fc4c:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1307.00 at time 2024-07-24 08:52:58.354932183 +0000 UTC m=+10936.376141241 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1307.00 at time 2024-07-24 08:52:58.354932183 +0000 UTC m=+10936.376141241 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2354).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 01e09cbd72ec2e2e35ea0ba190eb4931fc1e19e7:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 318.00 at time 2024-07-26 16:12:25.515485016 +0000 UTC m=+37305.407165096 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 318.00 at time 2024-07-26 16:12:25.515485016 +0000 UTC m=+37305.407165096 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2354).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 742a2251a05088a238fc6eb4c2466dace3cb1bb2:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 42.00 at time 2024-07-27 09:59:57.076328189 +0000 UTC m=+15006.073961004 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 42.00 at time 2024-07-27 09:59:57.076328189 +0000 UTC m=+15006.073961004 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 742a2251a05088a238fc6eb4c2466dace3cb1bb2:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1504.00 at time 2024-07-28 09:10:23.262604418 +0000 UTC m=+12068.962587162 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1504.00 at time 2024-07-28 09:10:23.262604418 +0000 UTC m=+12068.962587162 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2354).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 742a2251a05088a238fc6eb4c2466dace3cb1bb2:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 5.00 at time 2024-07-29 08:50:25.601690235 +0000 UTC m=+10730.281728471 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 5.00 at time 2024-07-29 08:50:25.601690235 +0000 UTC m=+10730.281728471 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ c754434cecbdc6016bc6cc6b7541ffbd71a50bed:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1166.00 at time 2024-07-30 09:53:02.185306068 +0000 UTC m=+14589.352957484 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:594
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1166.00 at time 2024-07-30 09:53:02.185306068 +0000 UTC m=+14589.352957484 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 05f80779a55122c949edecd7069abee4e6597f48:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1620.00 at time 2024-08-01 08:54:26.587123791 +0000 UTC m=+11164.030443421 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:594
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1620.00 at time 2024-08-01 08:54:26.587123791 +0000 UTC m=+11164.030443421 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2354).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 46c11ab2cf6fa6f6adf7abb99ef72f68e4d538f8:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1685.00 at time 2024-08-02 06:34:41.246488517 +0000 UTC m=+2922.333261539 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 89c848692fd0888dfbc891e52806b8b9c7ddd778:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1671.00 at time 2024-08-03 06:38:22.373458257 +0000 UTC m=+2996.580757162 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 8771ff1be0d425de9810b03d5ce1e474fa264b0b:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1672.00 at time 2024-08-04 06:06:45.948462158 +0000 UTC m=+1181.852123528 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 8771ff1be0d425de9810b03d5ce1e474fa264b0b:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1553.00 at time 2024-08-05 06:07:29.06830285 +0000 UTC m=+1282.748912092 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:588
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:605
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1553.00 at time 2024-08-05 06:07:29.06830285 +0000 UTC m=+1282.748912092 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2434).Run: context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 456fc36c2e2447bba18f80fba6be8a723ff03982:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1672.00 at time 2024-08-06 11:06:18.59595945 +0000 UTC m=+19170.374967464 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 33aa104f3337fdf02497dd90382d2cb9331ddb5e:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1674.00 at time 2024-08-07 11:37:59.772130967 +0000 UTC m=+21090.207006823 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 month ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ e675285bd6105c79fa14f20c060d0b355c8d2b57:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1673.00 at time 2024-08-08 10:26:26.904963181 +0000 UTC m=+16713.122136127 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 831797a28be6428dce2616fbe3eda09a25fc4e8b:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1670.00 at time 2024-08-09 10:01:57.480763241 +0000 UTC m=+15234.930688567 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ b8b31aaa48e6ab9e4d4c301e1c5a739901cb72b8:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1676.00 at time 2024-08-10 12:51:41.688701787 +0000 UTC m=+25396.919471747 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ b8b31aaa48e6ab9e4d4c301e1c5a739901cb72b8:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1673.00 at time 2024-08-11 07:13:19.124869582 +0000 UTC m=+5228.973210917 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ b8b31aaa48e6ab9e4d4c301e1c5a739901cb72b8:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1674.00 at time 2024-08-12 10:43:58.927768072 +0000 UTC m=+17799.325808358 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ db3f6e13d70f2b092e3c726ba3452c0b3aaea1ce:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1673.00 at time 2024-08-13 09:58:04.166593121 +0000 UTC m=+14931.407181844 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 191522a802ca853ed6776fc18d8c4d8f38599bbb:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1673.00 at time 2024-08-14 09:20:44.727876426 +0000 UTC m=+12738.425098617 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 2a5af58bede2d8cb9782250f57ed368e0f303dd9:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1673.00 at time 2024-08-15 09:35:42.90438177 +0000 UTC m=+13607.116772745 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 3d86d68d053783489fe34b454abe8f099dfd72a9:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1672.00 at time 2024-08-16 12:20:12.868741891 +0000 UTC m=+15135.851080611 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ ad78dc73db84c9749fd5d8da982bab8030376251:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1674.00 at time 2024-08-17 07:05:19.376307499 +0000 UTC m=+4779.265965006 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ ad78dc73db84c9749fd5d8da982bab8030376251:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1673.00 at time 2024-08-18 06:58:39.910324505 +0000 UTC m=+4328.948733906 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ ad78dc73db84c9749fd5d8da982bab8030376251:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1671.00 at time 2024-08-19 10:25:16.40839284 +0000 UTC m=+16732.207434807 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 9b56fdc97c3c8c183901ec7cbc67ffab958ef62d:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1672.00 at time 2024-08-20 10:11:15.616018943 +0000 UTC m=+15746.490036294 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ aa5cfc570c2875548db61118978d6c293516f908:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1672.00 at time 2024-08-21 07:00:05.04786094 +0000 UTC m=+4379.396382550 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 weeks ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.2 @ 310388de99f5b0b708a6b286581d79bb4df6f34d:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1672.00 at time 2024-08-22 07:01:40.350500537 +0000 UTC m=+4351.586568662 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #125731 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

This test on roachdash | Improve this report!