ZcashFoundation / zebra

Zcash - Financial Privacy in Rust 🦓
https://zfnd.org/zebra/
Apache License 2.0
408 stars 100 forks source link

Spurious Test Failure: batch_flushes_on_max_latency #985

Closed yaahc closed 3 years ago

yaahc commented 4 years ago

Description

It looks like the above test failed spuriously in CI when generating coverage here: https://github.com/ZcashFoundation/zebra/runs/1058412290?check_suite_focus=true

We should act pretty aggressively to address spurious failures imo. In this case it might be as simple as increasing a timeout to accommodate worst case latencies in CI, baring that I think we should disable this test until we can come up with a version that doesn't experience spurious failures.

[INFO tarpaulin] Launching test
[INFO tarpaulin] running /home/runner/work/zebra/zebra/target/debug/deps/ed25519-5d5008d14159b234

running 3 tests
test batch_flushes_on_max_items ... ok
test fallback_verification ... ok
test batch_flushes_on_max_latency ... FAILED

failures:

---- batch_flushes_on_max_latency stdout ----
Error: 
   0: deadline has elapsed

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
The application panicked (crashed).
Message:  assertion failed: `(left == right)`
  left: `1`,
 right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure
Location: /rustc/04488afe34512aa4c33566eb16d8c912a3ae04f9/src/libstd/macros.rs:16

Backtrace omitted.
Run with RUST_BACKTRACE=1 environment variable to display it.
teor2345 commented 4 years ago

In my experience, heavily loaded CI builders can take 5-10 seconds longer than expected to get enough CPU time to make progress.

So let's change the test timeout to at least 5 seconds? https://github.com/ZcashFoundation/zebra/blob/ad0001f7f7a1a35030cd6c5e6036ec9d27c4ebe6/tower-batch/tests/ed25519.rs#L149

teor2345 commented 3 years ago

I haven't seen this failure in months, so I think we can close this ticket without making any changes.