search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.12k stars 3.81k forks source link

roachtest: kv/gracefuldraining failed #125731

Closed cockroach-teamcity closed 2 months ago

cockroach-teamcity commented 5 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 3aa5bdf40527e2f60b179094403b4302a1c2cbe1:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:590
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 5.00 at time 2024-06-15 08:46:06.251665142 +0000 UTC m=+10554.934669970 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:573
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:590
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 5.00 at time 2024-06-15 08:46:06.251665142 +0000 UTC m=+10554.934669970 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-39576

kvoli commented 4 months ago

The workload exits with an error result is ambiguous early:

Error: ERROR: result is ambiguous: error=ba: Put [/Table/106/1/-8592365221218082830/0], EndTxn(parallel commit) [/Table/106/1/-8592365221218082830/0], [txn: 6d018e98], [can-forward-ts] RPC error: grpc: connection error: desc = "transport: error while dialing: connection interrupted (did the remote node shut down or are there networking issues?)" [code 14/Unavailable] [exhausted] (last error: ba: Put [/Table/106/1/-8592365221218082830/0], EndTxn(parallel commit) [/Table/106/1/-8592365221218082830/0], [txn: 6d018e98], [can-forward-ts] RPC error: grpc: connection error: desc = "transport: error while dialing: connection interrupted (did the remote node shut down or are there networking issues?)" [code 14/Unavailable]) (SQLSTATE 40003)

This causes QPS to drop to zero shortly after the node is stopped, as there QPS inbound is now zero.

2024/06/15 08:46:04 cluster.go:692: test status: stopping nodes :6
teamcity-15670966-1718430614-34-n7cpu4: stopping
2024/06/15 08:46:05 kv.go:565: test worker status: workload command completed

The ambiguous error seems expected, or at least always possible with nodes being stopped? Presumably, the lease transferred away to another node but the EndTxn was issued just prior, the node shuts down and the error is returned to the client?

arulajmani commented 4 months ago

The ambiguous error seems expected, or at least always possible with nodes being stopped? Presumably, the lease transferred away to another node but the EndTxn was issued just prior, the node shuts down and the error is returned to the client?

This sounds right. Does that mean all we need to do here is handle ambiguous errors and not exit the workload?

Either way, this shouldn't be a release blocker.

cockroach-teamcity commented 4 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 9e1cd533828b7887a48db8635f705669287cf2d6:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 618.00 at time 2024-07-02 06:41:16.603455958 +0000 UTC m=+3101.852501729 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 618.00 at time 2024-07-02 06:41:16.603455958 +0000 UTC m=+3101.852501729 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 7461a85f839c50edbdea502fe09794739a99632e:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 5.00 at time 2024-07-03 07:10:05.06479955 +0000 UTC m=+4828.596983709 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 5.00 at time 2024-07-03 07:10:05.06479955 +0000 UTC m=+4828.596983709 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 8f72a01e81d4751c6615a9a864b8e32076565b09:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 440.00 at time 2024-07-04 06:25:27.922898124 +0000 UTC m=+2154.303250354 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 440.00 at time 2024-07-04 06:25:27.922898124 +0000 UTC m=+2154.303250354 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ c557fb59f6aec659d364e9002fc083c59c6392b6:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 378.00 at time 2024-07-06 06:53:33.961372412 +0000 UTC m=+3831.626454968 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 378.00 at time 2024-07-06 06:53:33.961372412 +0000 UTC m=+3831.626454968 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 485975b3a824c68c07340e6a336c7864c00d3c6d:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1719.00 at time 2024-07-07 06:44:20.543825105 +0000 UTC m=+3261.381461868 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1719.00 at time 2024-07-07 06:44:20.543825105 +0000 UTC m=+3261.381461868 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 485975b3a824c68c07340e6a336c7864c00d3c6d:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1012.00 at time 2024-07-08 06:33:03.43094553 +0000 UTC m=+2628.098224269 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1012.00 at time 2024-07-08 06:33:03.43094553 +0000 UTC m=+2628.098224269 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 6b22014dec8c5e8e6006d710ed05bab5b01b4ae2:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 431.00 at time 2024-07-09 06:43:49.570197572 +0000 UTC m=+3311.918286188 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 431.00 at time 2024-07-09 06:43:49.570197572 +0000 UTC m=+3311.918286188 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 0b918d1dc3a9ce1f04975202be3b04ec375a816e:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 5.00 at time 2024-07-10 07:02:19.785659698 +0000 UTC m=+4395.709870339 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 5.00 at time 2024-07-10 07:02:19.785659698 +0000 UTC m=+4395.709870339 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ b78574b4b75b6f9ab8841d96f350697030f5e4df:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1624.00 at time 2024-07-11 06:30:09.47868389 +0000 UTC m=+2406.556126701 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1624.00 at time 2024-07-11 06:30:09.47868389 +0000 UTC m=+2406.556126701 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2354).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 3afb6935d0ef3de7c0d44cfb3cd54f312752c186:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 762.00 at time 2024-07-12 06:37:57.797786958 +0000 UTC m=+2799.762398116 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:579
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:596
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 762.00 at time 2024-07-12 06:37:57.797786958 +0000 UTC m=+2799.762398116 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv branch-release-24.2 release-blocker]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ dc3ebba01015d2c7275aa27af2a54fe88b87e901:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 93.00 at time 2024-07-13 06:50:42.395884569 +0000 UTC m=+3708.061135700 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:594
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 93.00 at time 2024-07-13 06:50:42.395884569 +0000 UTC m=+3708.061135700 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ bcd4e5e37f2a95e73347f5ed83db5a24bebfd8d1:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 763.00 at time 2024-07-17 11:00:24.062669044 +0000 UTC m=+18628.684717476 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:594
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 763.00 at time 2024-07-17 11:00:24.062669044 +0000 UTC m=+18628.684717476 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 46854026bc11b3870c8bbefc976f5550b18dd82c:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1743.00 at time 2024-07-18 09:57:34.581588925 +0000 UTC m=+14774.939987843 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:594
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1743.00 at time 2024-07-18 09:57:34.581588925 +0000 UTC m=+14774.939987843 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 59be0708627bb27f49d8976182062c6886d64282:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1682.00 at time 2024-07-19 10:14:40.481279883 +0000 UTC m=+15881.149445120 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:594
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1682.00 at time 2024-07-19 10:14:40.481279883 +0000 UTC m=+15881.149445120 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 05052df0b4d89dd1147714431163cd8a9b63c589:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1426.00 at time 2024-07-21 10:11:45.879313505 +0000 UTC m=+15598.811505636 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:594
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1426.00 at time 2024-07-21 10:11:45.879313505 +0000 UTC m=+15598.811505636 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 05052df0b4d89dd1147714431163cd8a9b63c589:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 336.00 at time 2024-07-22 09:59:10.923322292 +0000 UTC m=+14855.595089367 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:594
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 336.00 at time 2024-07-22 09:59:10.923322292 +0000 UTC m=+14855.595089367 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 7c29b0c71f3ee8ccc436b5f039955bf6d2add914:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1055.00 at time 2024-07-23 09:58:55.271566202 +0000 UTC m=+14823.358292890 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:594
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1055.00 at time 2024-07-23 09:58:55.271566202 +0000 UTC m=+14823.358292890 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ da85c982fa391ef34fd14c29006b1f629217ff42:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1330.00 at time 2024-07-25 06:11:56.515000813 +0000 UTC m=+1200.398091495 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:594
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1330.00 at time 2024-07-25 06:11:56.515000813 +0000 UTC m=+1200.398091495 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2393).Stop: cluster.StopE: one or more parallel execution failure(s): context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 7fb362dd5aa6e85d65c4c89f208c5bed51ab9692:

(assertions.go:363).Fail: 
    Error Trace:    github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                                main/pkg/cmd/roachtest/monitor.go:120
                                golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 224.00 at time 2024-07-26 10:36:08.006332969 +0000 UTC m=+17076.016849562 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.2
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:594
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.4
                      |     github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests/kv.go:611
                      | main.(*monitorImpl).Go.func1
                      |     main/pkg/cmd/roachtest/monitor.go:120
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     golang.org/x/sync/errgroup/external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 224.00 at time 2024-07-26 10:36:08.006332969 +0000 UTC m=+17076.016849562 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 1893ac1d7188dc87dbb2b379b30172bf83ca4645:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1671.00 at time 2024-07-27 07:03:03.291185491 +0000 UTC m=+4270.283956418 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ fb8032d12d2a787b54076812595e9d7fb22f925a:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1672.00 at time 2024-07-28 06:42:29.219087922 +0000 UTC m=+3082.647654968 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ fb8032d12d2a787b54076812595e9d7fb22f925a:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1670.00 at time 2024-07-29 08:37:31.994055758 +0000 UTC m=+9987.953130402 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 5d9405ff2e16622c055f94208992a5347c4c273e:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1674.00 at time 2024-07-30 08:19:15.510122973 +0000 UTC m=+8906.249544314 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ c7d1ceed692d36e9d7c5a71e7122408e2e3e3b8b:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1671.00 at time 2024-07-31 07:33:23.966222975 +0000 UTC m=+6274.971988889 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 7a1bf52daa3f3dcaa434fd66d2b3c960a3c9c2a5:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1672.00 at time 2024-08-01 07:22:59.883623297 +0000 UTC m=+5725.420751832 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 78fa126dfc749e620c28531c1659e35bf98034ec:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1673.00 at time 2024-08-02 07:43:12.116720599 +0000 UTC m=+6959.989779787 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

andrewbaptist commented 3 months ago

I think there are 2 separate issues here. The ambiguous write error which could be caused by a complex interaction of a range split and a node shutdown happening. This was fixed by #121691 (and has been backported to 24.1 and 24.2). There is a second issue which is the drop in QPS that I'm still investigating. That appears to be a change in behavior and in many of the failures it occurs before the shutdown even happens.

Something appears to be significantly impacting latency/throughput even at a low cluster utilization.

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 356e1cc24a67d7e9a1e637b98d1664794f70fa03:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1685.00 at time 2024-08-03 07:50:00.613848925 +0000 UTC m=+7290.086711230 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ a0a52309cbbe5cb68d576fb366411fa7d7cf3574:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1671.00 at time 2024-08-04 07:02:52.243697228 +0000 UTC m=+4522.152499374 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 5f7709cf41a3996c4bfd7eae1e231e75b024db47:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1668.00 at time 2024-08-05 06:54:14.38338034 +0000 UTC m=+3974.556785071 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ dbc62db07a1f149d305cff463b350588086f1b08:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:588: QPS of 1673.00 at time 2024-08-06 07:22:34.038685349 +0000 UTC m=+5728.219030644 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 7917e91eadb2f622faadf635a3e13c384e42070e:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1675.00 at time 2024-08-07 07:50:04.6370773 +0000 UTC m=+7294.171213510 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 533174dc4d87c792722bae844642f42dcf149aa7:

(soon.go:64).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1673.00 at time 2024-08-08 07:40:02.736016273 +0000 UTC m=+6714.262490885 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 2558f3f3a1dab0907db3dafd510ec5bc2464073f:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1672.00 at time 2024-08-09 06:54:22.112592341 +0000 UTC m=+4005.128275498 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 27468a22f82fb1e500bf7b35a62ca58d80abac39:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1672.00 at time 2024-08-10 06:52:45.038052647 +0000 UTC m=+3923.476960373 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 451cdbe687b3303665bfd490af59c5df777c8909:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1672.00 at time 2024-08-11 06:57:09.569125207 +0000 UTC m=+4221.642280625 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 2f0b0399a5c9656bdf8a417d0ab57e4eb1ae0f64:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1671.00 at time 2024-08-12 07:35:50.395362842 +0000 UTC m=+6523.156756790 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 9765a2cc0d7b4d6dacff07ea5668d1b59a36cfc2:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1674.00 at time 2024-08-13 07:23:23.078251961 +0000 UTC m=+5646.113586455 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 3 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ abe3d27bee5120afa287df2f8cc81c82ab6864da:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1672.00 at time 2024-08-14 07:08:28.590501379 +0000 UTC m=+4796.642143256 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 575cdd4696dfcac8f311d1ea546683271102f73e:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1671.00 at time 2024-08-15 07:02:17.314527318 +0000 UTC m=+4419.616751498 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ ec66ae778100724b11f1b8978c915be1bcea390c:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1669.00 at time 2024-08-16 07:13:50.981056569 +0000 UTC m=+4986.600669806 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/cpu_arch=arm64/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 1993fc04b5116f20f4814d637c7ce87b003687e4:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1673.00 at time 2024-08-17 07:15:09.231536146 +0000 UTC m=+5248.756283644 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 1993fc04b5116f20f4814d637c7ce87b003687e4:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1670.00 at time 2024-08-18 07:44:16.514101361 +0000 UTC m=+7021.395159002 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 1993fc04b5116f20f4814d637c7ce87b003687e4:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1673.00 at time 2024-08-19 07:04:26.540916876 +0000 UTC m=+4536.913249267 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 08bb256bc15218deaa8884bc89bddb2766514598:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1673.00 at time 2024-08-19 08:10:17.75355499 +0000 UTC m=+5394.940677569 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ 08bb256bc15218deaa8884bc89bddb2766514598:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1672.00 at time 2024-08-19 22:46:14.718575807 +0000 UTC m=+18135.221819762 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ ba7c94a6ffe5f0a990000188ecfd3a23f7c8b4a6:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1673.00 at time 2024-08-20 07:06:12.777629934 +0000 UTC m=+4650.007203466 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 months ago

roachtest.kv/gracefuldraining failed with artifacts on master @ f1a5aa8a633636138fededdf46a8dcc20f899a48:

(soon.go:65).SucceedsWithin: condition failed to evaluate within 45s: from kv.go:581: QPS of 1672.00 at time 2024-08-21 07:09:03.32442901 +0000 UTC m=+4734.340964541 is below minimum allowable QPS of 1800.00
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #128168 roachtest: kv/gracefuldraining failed [A-testing C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2.0-rc] - #126986 roachtest: kv/gracefuldraining failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv X-duplicate branch-release-24.2]

This test on roachdash | Improve this report!