roachtest: admission-control/disk-bandwidth-limiter failed

cockroachdb / cockroach

CockroachDB — the cloud native, distributed SQL database designed for high availability, effortless scale, and control over data placement.

https://www.cockroachlabs.com

Other

30.06k stars 3.8k forks source link

roachtest: admission-control/disk-bandwidth-limiter failed #132977

Open cockroach-teamcity opened 6 days ago

cockroach-teamcity commented 6 days ago

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on release-24.3 @ 4cbedefd790c75cb0f21f77ed8d917c8528a7d15:

(admission_control_disk_bandwidth_overload.go:185).3: write + read bandwidth 79.985000 (65.144219 + 14.840781) exceeded threshold of 78.750000
(cluster.go:2449).Run: context canceled
(cluster.go:2449).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131484 roachtest: admission-control/disk-bandwidth-limiter failed [A-storage C-test-failure O-roachtest O-robot P-1 T-storage branch-master]

/cc @cockroachdb/storage

This test on roachdash | Improve this report!

Jira issue: CRDB-43376

cockroach-teamcity commented 5 days ago

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on release-24.3 @ e466862ab36572bdc894cec84536b45472bdeb5b:

(admission_control_disk_bandwidth_overload.go:185).3: write + read bandwidth 81.406797 (68.554531 + 12.852266) exceeded threshold of 78.750000
(cluster.go:2449).Run: context canceled
(cluster.go:2449).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131484 roachtest: admission-control/disk-bandwidth-limiter failed [A-storage C-test-failure O-roachtest O-robot P-1 T-storage branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 4 days ago

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on release-24.3 @ b2d2353b876af1748607e155ecdfed9d4bba29d3:

(admission_control_disk_bandwidth_overload.go:185).3: write + read bandwidth 80.623437 (68.173594 + 12.449844) exceeded threshold of 78.750000
(cluster.go:2449).Run: context canceled
(cluster.go:2449).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131484 roachtest: admission-control/disk-bandwidth-limiter failed [A-storage C-test-failure O-roachtest O-robot P-1 T-storage branch-master]

This test on roachdash | Improve this report!

aadityasondhi commented 4 days ago

Same issue as https://github.com/cockroachdb/cockroach/issues/131484.

cockroach-teamcity commented 2 days ago

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on release-24.3 @ f4bf28879d49979d438967379b44c2fe3464f968:

(admission_control_disk_bandwidth_overload.go:133).func1: failed to set kvadmission.store.provisioned_bandwidth: dial tcp 18.119.158.141:26257: connect: connection refused
(cluster.go:2449).Run: context canceled
(cluster.go:2449).Run: context canceled
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/cpu_arch=arm64/run_1

Parameters:

ROACHTEST_arch=arm64
ROACHTEST_cloud=aws
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

Grafana is not yet available for aws clusters

Same failure on other branches

- #131484 roachtest: admission-control/disk-bandwidth-limiter failed [A-storage C-test-failure O-roachtest O-robot P-1 T-storage branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 2 days ago

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on release-24.3 @ f4bf28879d49979d438967379b44c2fe3464f968:

(admission_control_disk_bandwidth_overload.go:185).3: write + read bandwidth 80.070234 (67.631641 + 12.438594) exceeded threshold of 78.750000
(cluster.go:2449).Run: context canceled
(cluster.go:2449).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131484 roachtest: admission-control/disk-bandwidth-limiter failed [A-storage C-test-failure O-roachtest O-robot P-1 T-storage branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 1 day ago

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on release-24.3 @ 36f5b311f33775bbeb26d56003f6831ec9ddd837:

(admission_control_disk_bandwidth_overload.go:185).3: write + read bandwidth 80.322031 (67.695781 + 12.626250) exceeded threshold of 78.750000
(cluster.go:2449).Run: context canceled
(cluster.go:2449).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131484 roachtest: admission-control/disk-bandwidth-limiter failed [A-storage C-test-failure O-roachtest O-robot P-1 T-storage branch-master]

This test on roachdash | Improve this report!

cockroach-teamcity commented 19 hours ago

roachtest.admission-control/disk-bandwidth-limiter failed with artifacts on release-24.3 @ 047a8e99eb7a35e3f13ecf6b113f1965f206e3b9:

(admission_control_disk_bandwidth_overload.go:185).3: write + read bandwidth 82.740859 (67.816016 + 14.924844) exceeded threshold of 78.750000
(cluster.go:2449).Run: context canceled
(cluster.go:2449).Run: context canceled
(monitor.go:149).Wait: monitor failure: monitor user task failed: t.Fatal() was called
test artifacts and logs in: /artifacts/admission-control/disk-bandwidth-limiter/run_1

Parameters:

ROACHTEST_arch=amd64
ROACHTEST_cloud=gce
ROACHTEST_coverageBuild=false
ROACHTEST_cpu=8
ROACHTEST_encrypted=false
ROACHTEST_fs=ext4
ROACHTEST_localSSD=true
ROACHTEST_runtimeAssertionsBuild=false
ROACHTEST_ssd=0
Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

Same failure on other branches

- #131484 roachtest: admission-control/disk-bandwidth-limiter failed [A-storage C-test-failure O-roachtest O-robot P-1 T-storage branch-master]

This test on roachdash | Improve this report!