Open reubenmiller opened 1 month ago
https://github.com/thin-edge/thin-edge.io/pull/3027 introduced running the Rust tests using cargo-nextest which allows marking specific tests as flaky so they are automatically retried on failure, as this will hopefully be a reliable workaround (as the fixed sleep workaround also sporadically failed).
Below is the new console output showing the flakiness of the bridge_reconnects_successfully_after_local_connection_interrupted
test:
Caused by:
0: timed-out waiting for received message
1: deadline has elapsed
RETRY 2/5 [ ] tedge_mqtt_bridge::bridge bridge_reconnects_successfully_after_local_connection_interrupted
TRY 2 PASS [ 0.264s] tedge_mqtt_bridge::bridge bridge_reconnects_successfully_after_local_connection_interrupted
------------
Summary [ 31.118s] 1294 tests run: 1294 passed (1 flaky, 1 leaky), 3 skipped
FLAKY 2/5 [ 0.264s] tedge_mqtt_bridge::bridge bridge_reconnects_successfully_after_local_connection_interrupted
Finished report saved to codecov.json
Reference CI Run:
Describe the bug
Flaky unit tests:
bridge_reconnects_successfully_after_local_connection_interrupted
.bridge_reconnects_successfully_after_cloud_connection_interrupted
(seemingly related) - Marked as flaky in https://github.com/thin-edge/thin-edge.io/pull/3029Though a workaround has been added in a PR, which adds a sleep before interrupting the connection.
Fail
Below shows an example when the test fails.
Pass
Below shows an example when the test passes.
Notes
To Reproduce
A flaky test can be reproduced by re-running the tests until it fails. Sometimes it takes more than ~30 attempts
Expected behavior
The test should not fail sporadically.
Screenshots
Environment (please complete the following information):
MacOS
andUbuntu 22.04
aarch64
andx86_64
1.1.2~306+gfae4a56
Additional context