roachtest: rebalance/by-load/replicas/mixed-version failed

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 18c4e23306bb99cc28c83bb3e5726f6f4279b798:

(mixedversion.go:710).Run: mixed-version test failure while running step 15 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=99.2 tolerance=20.0% (±19.8) bounds=[79.3, 119.0]
    below  = []
    within = [s1: 85 (-14.2%), s2: 86 (-12.4%), s3: 88 (-10.5%), s5: 80 (-18.6%), s6: 85 (-13.6%)]
    above  = [s4: 167 (+69.3%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-41554

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 9e48858c6c8b22af4ec1159bcff6e233e7bfddff:

(mixedversion.go:710).Run: mixed-version test failure while running step 20 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=154.4 tolerance=20.0% (±30.9) bounds=[123.5, 185.3]
    below  = [s3: 79 (-48.4%)]
    within = [s1: 171 (+11.1%), s2: 166 (+7.6%), s4: 168 (+9.0%), s5: 172 (+11.7%), s6: 168 (+8.9%)]
    above  = []
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

[tbg]

@andrewbaptist could you share your thinking here? It makes me nervous when we remove release blocker labels without explanation. Thanks!

[andrewbaptist]

This was introduced by the test change to enable shared process deployments (so at least not a regression). We do still need to track down and fix.

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 3ab493e264e790d18ee80a3ae013eca00a023fa4:

(mixedversion.go:710).Run: mixed-version test failure while running step 35 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=102.5 tolerance=20.0% (±20.5) bounds=[82.0, 123.0]
    below  = [s3: 81 (-20.7%), s5: 65 (-36.5%)]
    within = [s2: 116 (+14.0%), s4: 92 (-9.7%), s6: 88 (-13.2%)]
    above  = [s1: 170 (+66.1%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 3ab493e264e790d18ee80a3ae013eca00a023fa4:

(mixedversion.go:710).Run: mixed-version test failure while running step 23 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=119.7 tolerance=20.0% (±23.9) bounds=[95.8, 143.6]
    below  = [s2: 88 (-25.9%), s3: 83 (-30.5%), s5: 57 (-52.3%)]
    within = []
    above  = [s1: 172 (+43.8%), s4: 167 (+39.9%), s6: 149 (+25.0%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ dafb6dd507b38fb3d6eb8b7e2493c7b8abed34d2:

(mixedversion.go:710).Run: mixed-version test failure while running step 12 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=141.0 tolerance=20.0% (±28.2) bounds=[112.8, 169.2]
    below  = [s2: 85 (-39.7%), s3: 87 (-37.8%)]
    within = [s1: 166 (+18.4%), s4: 165 (+17.7%), s5: 167 (+19.1%)]
    above  = [s6: 172 (+22.3%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 8551145a0c99c4c95a28ec470e699d0c20ca97ab:

(mixedversion.go:710).Run: mixed-version test failure while running step 15 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=98.7 tolerance=20.0% (±19.7) bounds=[78.9, 118.4]
    below  = [s1: 78 (-20.6%)]
    within = [s2: 90 (-8.6%), s3: 85 (-13.1%), s5: 82 (-16.6%), s6: 85 (-13.1%)]
    above  = [s4: 169 (+71.9%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ f341fb5d738192dbd9ea55e3fd7ed9d52407f6f5:

(mixedversion.go:710).Run: mixed-version test failure while running step 13 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=140.7 tolerance=20.0% (±28.1) bounds=[112.5, 168.8]
    below  = [s2: 80 (-42.9%), s6: 84 (-39.6%)]
    within = [s5: 165 (+17.8%)]
    above  = [s1: 169 (+20.8%), s3: 171 (+22.2%), s4: 171 (+21.8%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ fa9c0528fc0d06be1b4cfc534ec0501448111fbe:

(mixedversion.go:710).Run: mixed-version test failure while running step 36 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=132.8 tolerance=20.0% (±26.6) bounds=[106.2, 159.3]
    below  = [s1: 88 (-33.1%), s2: 92 (-30.3%), s4: 81 (-38.5%)]
    within = []
    above  = [s3: 170 (+28.6%), s5: 183 (+38.1%), s6: 179 (+35.2%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/cpu_arch=arm64/run_1

Parameters:

• ROACHTEST_arch=arm64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 5cc8013ada42f9ea03eda661a4f178c141f4f24d:

(mixedversion.go:710).Run: mixed-version test failure while running step 19 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=104.4 tolerance=20.0% (±20.9) bounds=[83.5, 125.3]
    below  = [s1: 82 (-20.7%), s6: 83 (-20.4%)]
    within = [s3: 118 (+13.6%), s4: 84 (-18.6%), s5: 86 (-17.5%)]
    above  = [s2: 170 (+63.5%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 1a8e39cd3c488c5c583b75b2f6a8337568de618c:

(mixedversion.go:710).Run: mixed-version test failure while running step 12 (restart node 4 with binary version v24.1.2): waiting for shared-process tenant on n4: pq: internal error while retrieving user account memberships: operation "get-user-session" timed out after 10.001s (given timeout 10s): internal error while retrieving user account: get auth info error: interrupted during singleflight load-value:authinfo-roachprod-2-2: context deadline exceeded
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 197c6ee5537ffb211ebd8dbcbe49edc6d5c710e1:

(mixedversion.go:720).Run: mixed-version test failure while running step 22 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=155.1 tolerance=20.0% (±31.0) bounds=[124.1, 186.1]
    below  = [s5: 81 (-47.5%)]
    within = [s1: 182 (+17.7%), s2: 158 (+2.2%), s3: 174 (+12.5%), s4: 166 (+7.4%), s6: 167 (+7.7%)]
    above  = []
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 197c6ee5537ffb211ebd8dbcbe49edc6d5c710e1:

(mixedversion.go:720).Run: mixed-version test failure while running step 25 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=98.6 tolerance=20.0% (±19.7) bounds=[78.9, 118.4]
    below  = [s1: 78 (-20.5%)]
    within = [s2: 84 (-14.4%), s4: 89 (-9.5%), s5: 84 (-14.6%), s6: 81 (-17.2%)]
    above  = [s3: 173 (+76.2%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ cba4e7f1fb6a3963c302cc82a58c42da67adc613:

(mixedversion.go:720).Run: mixed-version test failure while running step 11 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=156.9 tolerance=20.0% (±31.4) bounds=[125.6, 188.3]
    below  = [s2: 86 (-44.8%)]
    within = [s1: 164 (+5.0%), s3: 178 (+13.6%), s4: 166 (+5.9%), s5: 174 (+11.4%), s6: 170 (+9.0%)]
    above  = []
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=azure
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for azure clusters

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ c6122f6e6f0d35249a5eef8cab22db49dc43a626:

(mixedversion.go:720).Run: mixed-version test failure while running step 23 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=114.0 tolerance=20.0% (±22.8) bounds=[91.2, 136.8]
    below  = [s1: 84 (-26.0%), s2: 84 (-25.7%), s4: 83 (-26.5%), s6: 86 (-24.3%)]
    within = []
    above  = [s3: 172 (+51.1%), s5: 172 (+51.4%)]
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!

[cockroach-teamcity]

roachtest.rebalance/by-load/replicas/mixed-version failed with artifacts on master @ 83589fb87caa92fb42e83994f1691978f37e4cbb:

(cluster.go:2473).Run: full command output in run_113510.135459105_n7_cockroach-workload-i.log: COMMAND_PROBLEM: exit status 1
(mixedversion.go:720).Run: panic (stack trace above): t.Fatal() was called
test artifacts and logs in: /artifacts/rebalance/by-load/replicas/mixed-version/run_1

Parameters:

• ROACHTEST_arch=amd64
• ROACHTEST_cloud=gce
• ROACHTEST_coverageBuild=false
• ROACHTEST_cpu=4
• ROACHTEST_encrypted=false
• ROACHTEST_fs=ext4
• ROACHTEST_localSSD=true
• ROACHTEST_runtimeAssertionsBuild=false
• ROACHTEST_ssd=0
  Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

This test on roachdash | Improve this report!