davepacheco
commented
1 year ago This was not easy to reproduce locally and we don't have as much information as would be useful out of the failures. See #2995.
I did catch one with:
while RUST_BACKTRACE=1 cargo test -p omicron-nexus -- integration_tests::instances integration_tests::disks; do :; done
...
failures:
---- integration_tests::disks::test_disk_metrics stdout ----
log file: /dangerzone/omicron_tmp/test_all-842a3bf74e59f078-test_disk_metrics.18714.8.log
note: configured to log to "/dangerzone/omicron_tmp/test_all-842a3bf74e59f078-test_disk_metrics.18714.8.log"
thread 'integration_tests::disks::test_disk_metrics' panicked at 'Expected at least one measurement', nexus/tests/integration_tests/metrics.rs:34:5
stack backtrace:
0: rust_begin_unwind
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/std/src/panicking.rs:575:5
1: core::panicking::panic_fmt
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/panicking.rs:64:14
2: test_all::integration_tests::metrics::query_for_latest_metric::{{closure}}
at ./tests/integration_tests/metrics.rs:34:5
3: test_all::integration_tests::disks::test_disk_metrics::{{closure}}::test_disk_metrics::{{closure}}
at ./tests/integration_tests/disks.rs:1419:70
4: test_all::integration_tests::disks::test_disk_metrics::{{closure}}
at ./tests/integration_tests/disks.rs:1350:1
5: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/future/future.rs:125:9
6: <core::pin::Pin<P> as core::future::future::Future>::poll
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/future/future.rs:125:9
7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
at /home/dap/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.28.0/src/runtime/scheduler/current_thread.rs:541:57
8: tokio::runtime::coop::with_budget
at /home/dap/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.28.0/src/runtime/coop.rs:107:5
9: tokio::runtime::coop::budget
at /home/dap/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.28.0/src/runtime/coop.rs:73:5
10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
at /home/dap/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.28.0/src/runtime/scheduler/current_thread.rs:541:25
11: tokio::runtime::scheduler::current_thread::Context::enter
at /home/dap/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.28.0/src/runtime/scheduler/current_thread.rs:350:19
12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
at /home/dap/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.28.0/src/runtime/scheduler/current_thread.rs:540:36
13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
at /home/dap/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.28.0/src/runtime/scheduler/current_thread.rs:615:57
14: tokio::macros::scoped_tls::ScopedKey<T>::set
at /home/dap/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.28.0/src/macros/scoped_tls.rs:61:9
15: tokio::runtime::scheduler::current_thread::CoreGuard::enter
at /home/dap/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.28.0/src/runtime/scheduler/current_thread.rs:615:27
16: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
at /home/dap/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.28.0/src/runtime/scheduler/current_thread.rs:530:19
17: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
at /home/dap/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.28.0/src/runtime/scheduler/current_thread.rs:154:24
18: tokio::runtime::runtime::Runtime::block_on
at /home/dap/.cargo/registry/src/github.com-1ecc6299db9ec823/tokio-1.28.0/src/runtime/runtime.rs:302:47
19: test_all::integration_tests::disks::test_disk_metrics
at ./tests/integration_tests/disks.rs:1350:1
20: test_all::integration_tests::disks::test_disk_metrics::{{closure}}
at ./tests/integration_tests/disks.rs:1350:1
21: core::ops::function::FnOnce::call_once
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/ops/function.rs:250:5
22: core::ops::function::FnOnce::call_once
at /rustc/9eb3afe9ebe9c7d2b84b71002d44f4a0edac95e0/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: temporary directory leaked: /dangerzone/omicron_tmp/.tmpagosuV
WARN: dropped ClickHouseInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked)
WARN: dendrite temporary directory leaked: /dangerzone/omicron_tmp/.tmpaAr1Me
failures:
integration_tests::disks::test_disk_metrics
test result: FAILED. 45 passed; 1 failed; 0 ignored; 0 measured; 180 filtered out; finished in 67.45s
error: test failed, to rerun pass `-p omicron-nexus --test test_all`
I've seen a couple of apparently-spurious failures of test_instance_metrics and test_disk_metrics. Here's one from CI:
https://github.com/oxidecomputer/omicron/pull/2951/checks?check_run_id=13190550818 (buildomat job id 01GZFQ8S6NGARWQ8Z7HECHBTA2)