search

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.
https://www.cockroachlabs.com
Other
30.16k stars 3.82k forks source link

roachtest: rebalance/by-load/leases/mixed-version failed #135869

Open cockroach-teamcity opened 21 hours ago

cockroach-teamcity commented 21 hours ago

roachtest.rebalance/by-load/leases/mixed-version failed with artifacts on release-24.3 @ a8238bebce529c9fbd1c795a246fbcc7f96135e4:

(mixedversion.go:737).Run: mixed-version test failure while running step 11 (run "rebalance load run"): CPU not evenly balanced after timeout: outside bounds mean=18.3 tolerance=20.0% (±3.7) bounds=[14.6, 21.9]
    below  = [s3: 0 (-100.0%)]
    within = []
    above  = [s1: 29 (+60.3%), s2: 25 (+39.7%)]
test artifacts and logs in: /artifacts/rebalance/by-load/leases/mixed-version/run_1

Parameters:

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #132633 roachtest: rebalance/by-load/leases/mixed-version failed [A-testing C-bug C-test-failure O-roachtest O-robot P-2 T-kv branch-master]

/cc @cockroachdb/kv-triage

This test on roachdash | Improve this report!

Jira issue: CRDB-44755

kvoli commented 7 hours ago

Same as the known issue:

Details

``` 2024/11/21 06:58:13 rebalance_load.go:311: unable to get the cluster stores CPU: node idx 0 has core count normalized CPU utilization ts datapoint not in [0\%!,(float64=1.7340088639065105)100\{[{{%!](string=cr.node.sys.cpu.host.combined.percent-normalized) %!](*tspb.TimeSeriesQueryAggregator=0xc0081f0dac) %!](*tspb.TimeSeriesQueryAggregator=0xc0081f0ec0) %!](*tspb.TimeSeriesQueryDerivative=) [%!](string=1)] {%!](uint64=1)}} [{%!](int64=1732171910000000000) %!](float64=0.29299783958042985)} {%!](int64=1732171920000000000) %!](float64=1.7114446951781088)} {%!](int64=1732171930000000000) %!](float64=1.854075117871087)} {%!](int64=1732171940000000000) %!](float64=1.8539687914017469)} {%!](int64=1732171950000000000) %!](float64=1.8599500716862707)} {%!](int64=1732171960000000000) %!](float64=1.850513482656183)} {%!](int64=1732171970000000000) %!](float64=1.8274366689030337)} {%!](int64=1732171980000000000) %!](float64=1.753992296465834)} {%!](int64=1732171990000000000) %!](float64=1.677497445842389)} {%!](int64=1732172000000000000) %!](float64=1.7109844782910097)} {%!](int64=1732172010000000000) %!](float64=1.72096851161124)} {%!](int64=1732172020000000000) %!](float64=1.7209741778150465)} {%!](int64=1732172030000000000) %!](float64=1.73713138124204)} {%!](int64=1732172040000000000) %!](float64=1.7233753425743328)} {%!](int64=1732172050000000000) %!](float64=1.7200044192073543)} {%!](int64=1732172060000000000) %!](float64=1.7129714983811297)} {%!](int64=1732172070000000000) %!](float64=1.7570447537455076)} {%!](int64=1732172080000000000) %!](float64=1.7019907697636574)} {%!](int64=1732172090000000000) %!](float64=1.7154800640630794)} {%!](int64=1732172100000000000) %!](float64=1.729974961034404)} {%!](int64=1732172110000000000) %!](float64=1.7389689032363818)} {%!](int64=1732172120000000000) %!](float64=1.7114994112442026)} {%!](int64=1732172130000000000) %!](float64=1.6985092366630798)} {%!](int64=1732172140000000000) %!](float64=1.7579668628520253)} {%!](int64=1732172150000000000) %!](float64=1.734470970629706)} {%!](int64=1732172160000000000) %!](float64=1.716502932866911)} {%!](int64=1732172170000000000) %!](float64=1.7344970792803682)} {%!](int64=1732172180000000000) %!](float64=1.7134859533062006)} {%!](int64=1732172190000000000) %!](float64=1.7039867291809545)} {%!](int64=1732172200000000000) %!](float64=1.7404409058156602)} {%!](int64=1732172210000000000) %!](float64=1.7319609572701207)} {%!](int64=1732172220000000000) %!](float64=1.7388289347049355)} {%!](int64=1732172230000000000) %!](float64=1.6886714262506357)} {%!](int64=1732172240000000000) %!](float64=1.7146797992661353)} {%!](int64=1732172250000000000) %!](float64=1.7305170943939618)} {%!](int64=1732172260000000000) %!](float64=1.6999633668094232)} {%!](int64=1732172270000000000) %!](float64=1.7184772532039903)} {%!](int64=1732172280000000000) %!](float64=1.7340088639065105)}]} {{%!](string=cr.node.sys.cpu.host.combined.percent-normalized) %!](*tspb.TimeSeriesQueryAggregator=0xc0081f0ec4) %!](*tspb.TimeSeriesQueryAggregator=0xc0081f0ec8) %!](*tspb.TimeSeriesQueryDerivative=) [%!](string=2)] {%!](uint64=1)}} [{%!](int64=1732171910000000000) %!](float64=0.25549478404953313)} {%!](int64=1732171920000000000) %!](float64=1.2804815096409046)} {%!](int64=1732171930000000000) %!](float64=1.2685313403352942)} {%!](int64=1732171940000000000) %!](float64=1.2689619273352941)} {%!](int64=1732171950000000000) %!](float64=1.371969238803303)} {%!](int64=1732171960000000000) %!](float64=1.3425368860666909)} {%!](int64=1732171970000000000) %!](float64=1.3594518425919828)} {%!](int64=1732171980000000000) %!](float64=1.5998555730881394)} {%!](int64=1732171990000000000) %!](float64=1.6501249692443207)} {%!](int64=1732172000000000000) %!](float64=1.6569993547644513)} {%!](int64=1732172010000000000) %!](float64=1.6910015729696632)} {%!](int64=1732172020000000000) %!](float64=1.699426092129196)} {%!](int64=1732172030000000000) %!](float64=1.6902881157491112)} {%!](int64=1732172040000000000) %!](float64=1.6661155618726322)} {%!](int64=1732172050000000000) %!](float64=1.6996941117096855)} {%!](int64=1732172060000000000) %!](float64=1.7104606564971174)} {%!](int64=1732172070000000000) %!](float64=1.6783373080874593)} {%!](int64=1732172080000000000) %!](float64=1.6865008820399614)} {%!](int64=1732172090000000000) %!](float64=1.6879737574095883)} {%!](int64=1732172100000000000) %!](float64=1.6599822430039484)} {%!](int64=1732172110000000000) %!](float64=1.6804684263509246)} {%!](int64=1732172120000000000) %!](float64=1.6704738467273608)} {%!](int64=1732172130000000000) %!](float64=1.65850298431027)} {%!](int64=1732172140000000000) %!](float64=1.7184823692301328)} {%!](int64=1732172150000000000) %!](float64=1.6990169371600432)} {%!](int64=1732172160000000000) %!](float64=1.679481270256926)} {%!](int64=1732172170000000000) %!](float64=1.6705458429519677)} {%!](int64=1732172180000000000) %!](float64=1.627411969218055)} {%!](int64=1732172190000000000) %!](float64=1.6293428650335562)} {%!](int64=1732172200000000000) %!](float64=1.6471468192553076)} {%!](int64=1732172210000000000) %!](float64=1.6774460838634617)} {%!](int64=1732172220000000000) %!](float64=1.683004922789399)} {%!](int64=1732172230000000000) %!](float64=1.6714798364372891)} {%!](int64=1732172240000000000) %!](float64=1.6809963644591626)} {%!](int64=1732172250000000000) %!](float64=1.6455054739885098)} {%!](int64=1732172260000000000) %!](float64=1.6234868336841275)} {%!](int64=1732172270000000000) %!](float64=1.6303515154096637)} {%!](int64=1732172280000000000) %!](float64=1.6339861719018245)}]} {{%!](string=cr.node.sys.cpu.host.combined.percent-normalized) %!](*tspb.TimeSeriesQueryAggregator=0xc0081f0ecc) %!](*tspb.TimeSeriesQueryAggregator=0xc0081f0ef0) %!](*tspb.TimeSeriesQueryDerivative=) [%!](string=3)] {%!](uint64=1)}} [{%!](int64=1732171910000000000) %!](float64=0)} {%!](int64=1732171920000000000) %!](float64=0.6164769003023073)} {%!](int64=1732171930000000000) %!](float64=0.5977026835245317)} {%!](int64=1732171940000000000) %!](float64=0.6075461996964182)} {%!](int64=1732171950000000000) %!](float64=0.6207521779090162)} {%!](int64=1732171960000000000) %!](float64=0.6522192813849538)} {%!](int64=1732171970000000000) %!](float64=0.6522192813849538)} {%!](int64=1732171980000000000) %!](float64=0.8291387982478244)} {%!](int64=1732171990000000000) %!](float64=0.8699886118740706)} {%!](int64=1732172000000000000) %!](float64=0.8712382910801109)} {%!](int64=1732172010000000000) %!](float64=0.867510374209561)} {%!](int64=1732172020000000000) %!](float64=0.8694883304232197)} {%!](int64=1732172030000000000) %!](float64=0.8835033078363845)} {%!](int64=1732172040000000000) %!](float64=0.8835033078363845)} {%!](int64=1732172050000000000) %!](float64=0.8836567841454107)} {%!](int64=1732172060000000000) %!](float64=0.8865330594455821)} {%!](int64=1732172070000000000) %!](float64=0.8865330594455821)} {%!](int64=1732172080000000000) %!](float64=0.8754559883156763)} {%!](int64=1732172090000000000) %!](float64=0.8591049783328274)} {%!](int64=1732172100000000000) %!](float64=0.879968988220914)} {%!](int64=1732172110000000000) %!](float64=0.8764805444107635)} {%!](int64=1732172120000000000) %!](float64=0.871757907280098)} {%!](int64=1732172130000000000) %!](float64=0.8884998412250784)} {%!](int64=1732172140000000000) %!](float64=0.8904850525850528)} {%!](int64=1732172150000000000) %!](float64=0.8904850525850528)} {%!](int64=1732172160000000000) %!](float64=0.880675556482965)} {%!](int64=1732172170000000000) %!](float64=0.8698236450106226)} {%!](int64=1732172180000000000) %!](float64=0.8698236450106226)} {%!](int64=1732172190000000000) %!](float64=0.8824537179083825)} {%!](int64=1732172200000000000) %!](float64=0.8832444279642018)} {%!](int64=1732172210000000000) %!](float64=0.8832444279642018)} {%!](int64=1732172220000000000) %!](float64=0.8963183773674306)} {%!](int64=1732172230000000000) %!](float64=0.8963183773674306)} {%!](int64=1732172240000000000) %!](float64=0.8829845562469174)} {%!](int64=1732172250000000000) %!](float64=0.878245290058334)} {%!](int64=1732172260000000000) %!](float64=0.8769973983872177)} {%!](int64=1732172270000000000) %!](float64=0.8744993749078468)} {%!](int64=1732172280000000000) %!](float64=0.889738836090928)}]}]} (impossible!): %!v(MISSING) [resp=%!v(MISSING)] ```