search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.13k stars 3.81k forks source link

roachtest: kv/gracefuldraining failed #135161

Open cockroach-teamcity opened 1 day ago

cockroach-teamcity commented 1 day ago

roachtest.kv/gracefuldraining failed with artifacts on release-24.3.0-rc @ 516de012609eb9463b2df0fdc78958ca0549074e:

(assertions.go:363).Fail: 
    Error Trace:    pkg/cmd/roachtest/tests/kv.go:623
                                pkg/cmd/roachtest/monitor.go:115
                                external/org_golang_x_sync/errgroup/errgroup.go:78
                                src/runtime/asm_amd64.s:1695
    Error:          Received unexpected error:
                    QPS of 1792.00 at time 2024-11-14 11:38:39.835092719 +0000 UTC m=+17852.682065753 is below minimum allowable QPS of 1800.00
                    (1) attached stack trace
                      -- stack trace:
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.3
                      |     pkg/cmd/roachtest/tests/kv.go:606
                      | github.com/cockroachdb/cockroach/pkg/cmd/roachtest/tests.registerKVGracefulDraining.func1.5
                      |     pkg/cmd/roachtest/tests/kv.go:623
                      | main.(*monitorImpl).Go.func1
                      |     pkg/cmd/roachtest/monitor.go:115
                      | golang.org/x/sync/errgroup.(*Group).Go.func1
                      |     external/org_golang_x_sync/errgroup/errgroup.go:78
                      | runtime.goexit
                      |     src/runtime/asm_amd64.s:1695
                    Wraps: (2) QPS of 1792.00 at time 2024-11-14 11:38:39.835092719 +0000 UTC m=+17852.682065753 is below minimum allowable QPS of 1800.00
                    Error types: (1) *withstack.withStack (2) *errutil.leafError
    Test:           kv/gracefuldraining
(require.go:1357).NoError: FailNow called
(cluster.go:2451).Run: context canceled
test artifacts and logs in: /artifacts/kv/gracefuldraining/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #134044 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot T-kv X-duplicate branch-release-24.2.5-rc] - #133020 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-release-24.3] - #131569 roachtest: kv/gracefuldraining failed [C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master] - #130760 roachtest: kv/gracefuldraining failed [C-test-failure O-roachtest O-robot P-3 T-kv X-duplicate branch-release-24.2]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-44382

tbg commented 18 hours ago 
_elapsed___errors__ops/sec(inst)___ops/sec(cum)__p50(ms)__p95(ms)__p99(ms)_pMax(ms)
   62.0s        0          545.1          487.5      1.0      1.4      1.6      1.8 follower-read
   62.0s        0          479.1          485.0      1.0      1.5      1.7      4.7 read
   62.0s        0          974.3          974.4      2.2     37.7     48.2     67.1 write

   63.0s        0          496.0          487.7      1.0      1.4      1.5      1.8 follower-read
   63.0s        0          484.0          485.0      1.0      1.4      1.6      1.7 read
   63.0s        0         1021.0          975.2      2.2     17.8     25.2     28.3 write

   64.0s        0          519.7          488.2      1.0      1.4      1.6     10.0 follower-read
   64.0s        0          510.7          485.4      1.0      1.6      2.0      6.8 read
   64.0s        0          969.4          975.1      2.4     22.0     26.2     30.4 write

   65.0s        0          482.3          488.1      1.0      1.5      1.6      2.0 follower-read
   65.0s        0          510.4          485.8      1.0      1.5      1.6      2.4 read
   65.0s        0         1008.7          975.6      2.1     10.5     23.1     27.3 write

I don't see it, this looks fine (in an ideal world, write is 1000, and read and follower-read are 500). Removing rel-blocker.