roachtest: rebalance/by-load/leases/mixed-version failed

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

https://www.cockroachlabs.com

Other

30.05k stars 3.8k forks source link

roachtest: rebalance/by-load/leases/mixed-version failed #132633

Open cockroach-teamcity opened 1 week ago

cockroach-teamcity commented 1 week ago

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on master @ 49ca24cedb042579e9645c206640d59975805d12:

(mixedversion.go:732).Run: mixed-version test failure while running step 18 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=15.1 tolerance=20.0% (±3.0) bounds=[12.1, 18.1]
    below  = [s2: 5 (-62.9%), s3: 0 (-100.0%)]
    within = []
    above  = [s1: 39 (+162.9%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!

Jira issue: CRDB-43189}

cockroach-teamcity commented 5 days ago

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on master @ 42f40f59cae3c0fd8842e194d6991c951ab4382f:

(mixedversion.go:732).Run: mixed-version test failure while running step 11 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=34.4 tolerance=20.0% (±6.9) bounds=[27.5, 41.3]
    below  = [s2: 0 (-100.0%)]
    within = []
    above  = [s1: 47 (+37.8%), s3: 55 (+62.2%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

kvoli commented 4 days ago

This seems like a test issue:

https://grafana.testeng.crdb.io/d/J-yAVzkVddd/detailed?orgId=1&from=1729147186040&to=1729147989016&var-cluster=teamcity-17348947-1729144242-01-n4cpu4

We did recently change the CPU metric to be only gathered for the system tenant, I wonder if that caused unexpected issues.

cockroach-teamcity commented 1 day ago

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on master @ 472ea07a5232c98536293d13bb46cca59f9f2cd0:

(mixedversion.go:732).Run: mixed-version test failure while running step 18 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=17.9 tolerance=20.0% (±3.6) bounds=[14.3, 21.5]
    below  = [s2: 0 (-100.0%)]
    within = [s1: 16 (-9.9%)]
    above  = [s3: 37 (+109.9%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

_{This test on roachdash | Improve this report!}