roachtest: rebalance/by-load/leases/mixed-version failed

cockroach-teamcity commented 1 month ago

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on master @ f1a5aa8a633636138fededdf46a8dcc20f899a48:

(mixedversion.go:710).Run: mixed-version test failure while running step 18 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=115.4 tolerance=20.0% (±23.1) bounds=[92.3, 138.4]
    below  = [s3: 91 (-20.5%)]
    within = [s2: 96 (-16.5%)]
    above  = [s1: 158 (+37.0%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_metamorphicBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

/cc @cockroachdb/kv-triage _{This test on roachdash | Improve this report!

Jira issue: CRDB-41516}

renatolabs commented 1 month ago

FWIW, this failure also seems to happen only in shared-process deployments (which were added to this test in #129117). So it's kind of like #129031.

andrewbaptist commented 1 month ago

Caused by https://github.com/cockroachdb/cockroach/pull/129117 change. Test only issue.

cockroach-teamcity commented 1 month ago

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on master @ 9e48858c6c8b22af4ec1159bcff6e233e7bfddff:

(mixedversion.go:710).Run: mixed-version test failure while running step 16 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=111.1 tolerance=20.0% (±22.2) bounds=[88.9, 133.3]
    below  = [s2: 56 (-49.0%)]
    within = [s3: 92 (-16.7%)]
    above  = [s1: 184 (+65.7%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 1 month ago

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on master @ 7512f81be1591cde8e3b5d34bb0b9b1101f22d24:

(mixedversion.go:710).Run: mixed-version test failure while running step 13 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=123.3 tolerance=20.0% (±24.7) bounds=[98.6, 147.9]
    below  = [s2: 92 (-25.0%), s3: 93 (-24.1%)]
    within = []
    above  = [s1: 183 (+49.1%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 4 weeks ago

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on master @ fa9c0528fc0d06be1b4cfc534ec0501448111fbe:

(mixedversion.go:710).Run: mixed-version test failure while running step 12 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=120.0 tolerance=20.0% (±24.0) bounds=[96.0, 143.9]
    below  = [s1: 91 (-23.9%), s3: 89 (-25.1%)]
    within = []
    above  = [s2: 178 (+49.1%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 3 weeks ago

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on master @ 00edd9c46e7f3638846d0dc182a9dd50a0aa9cbc:

(mixedversion.go:710).Run: mixed-version test failure while running step 27 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=107.4 tolerance=20.0% (±21.5) bounds=[85.9, 128.8]
    below  = [s1: 49 (-54.4%), s2: 82 (-23.0%)]
    within = []
    above  = [s3: 190 (+77.3%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 3 weeks ago

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on master @ 1b0a374fd2a101cebcfb24cff4b3b57795ad1df6:

(mixedversion.go:710).Run: mixed-version test failure while running step 11 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=116.2 tolerance=20.0% (±23.2) bounds=[92.9, 139.4]
    below  = [s3: 90 (-21.7%)]
    within = [s2: 115 (-0.4%)]
    above  = [s1: 141 (+22.1%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 2 weeks ago

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on master @ 1a8e39cd3c488c5c583b75b2f6a8337568de618c:

(mixedversion.go:710).Run: mixed-version test failure while running step 12 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=121.9 tolerance=20.0% (±24.4) bounds=[97.5, 146.3]
    below  = [s1: 93 (-23.6%), s3: 94 (-22.3%)]
    within = []
    above  = [s2: 177 (+45.8%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=azure
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 2 weeks ago

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on master @ 1a8e39cd3c488c5c583b75b2f6a8337568de618c:

(mixedversion.go:710).Run: mixed-version test failure while running step 11 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=147.0 tolerance=20.0% (±29.4) bounds=[117.6, 176.4]
    below  = [s1: 92 (-37.4%)]
    within = [s3: 166 (+13.5%)]
    above  = [s2: 182 (+23.9%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroach-teamcity commented 2 weeks ago

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on master @ 197c6ee5537ffb211ebd8dbcbe49edc6d5c710e1:

(mixedversion.go:720).Run: mixed-version test failure while running step 12 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=116.7 tolerance=20.0% (±23.3) bounds=[93.4, 140.1]
    below  = [s1: 93 (-20.3%), s3: 85 (-26.9%)]
    within = []
    above  = [s2: 171 (+47.2%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=4
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=false
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

_{This test on roachdash | Improve this report!}

cockroachdb / cockroach

roachtest: rebalance/by-load/leases/mixed-version failed #129389