thin-edge / thin-edge.io

The open edge framework for lightweight IoT devices
https://thin-edge.io
Apache License 2.0
219 stars 54 forks source link

flaky unittest: tests::test_cli_pub_basic::some_2_expects #3043

Open reubenmiller opened 1 month ago

reubenmiller commented 1 month ago

Describe the bug

Flaky unit test.

Test name

tedge::mqtt tests::test_cli_pub_basic::some_2_expects

Failures

        FAIL [   0.637s] tedge::mqtt tests::test_cli_pub_basic::some_2_expects

--- STDOUT:              tedge::mqtt tests::test_cli_pub_basic::some_2_expects ---

running 1 test
test tests::test_cli_pub_basic::some_2_expects ... FAILED

failures:

failures:
    tests::test_cli_pub_basic::some_2_expects

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 3 filtered out; finished in 0.59s

--- STDERR:              tedge::mqtt tests::test_cli_pub_basic::some_2_expects ---
MQTT-TEST INFO: start test MQTT broker (port = 42667)
Ok(
    Ok(
        Incoming(
            ConnAck(
                ConnAck {
                    session_present: false,
                    code: Success,
                },
            ),
        ),
    ),
)
Ok(
    Ok(
        Outgoing(
            Subscribe(
                1,
            ),
        ),
    ),
)
Ok(
    Ok(
        Incoming(
            SubAck(
                SubAck {
                    pkid: 1,
                    return_codes: [
                        Success(
                            ExactlyOnce,
                        ),
                    ],
                },
            ),
        ),
    ),
)
thread 'tests::test_cli_pub_basic::some_2_expects' panicked at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/assert_cmd-2.0.13/src/output.rs:65:25:
command="/home/runner/work/thin-edge.io/thin-edge.io/target/llvm-cov-target/debug/tedge" "--config-dir" "/tmp/.tmpL1mtC1" "mqtt" "pub" "topic" "message" "--qos" "2"
code=1
stdout=""
stderr=```
ERROR: the message has not been published
Error: failed to publish the message \"message\" on the topic \"topic\" with QoS \"ExactlyOnce\".

Caused by:
    0: I/O: connection closed by peer
    1: connection closed by peer

note: run with RUST_BACKTRACE=1 environment variable to display a backtrace


**Build seen on**

* https://github.com/thin-edge/thin-edge.io/actions/runs/10199347399/job/28216228581

**To Reproduce**

Whilst it has not been able to be reproduced locally, you could run the following loop to stop on the first failure:

```bash
while just test-unit tests::test_cli_pub_basic::some_2_expects; do echo running tests; done

Expected behavior

The test should pass consistently.

rina23q commented 1 month ago

My doubt is this test is using std::process::Command, not tokio::process::Command.

However, I also have a doubt to the value of this test itself. It calls tedge binary, so it sounds that it's better to write such a test in system test level. Also, since the subscriber in the test has QoS 1 fixed, publishing with QoS 2 is downgraded to QoS 1. Actually, no assertion for QoS. So, I assume that this test's purpose is simply if tedge mqtt pub works and accepts --qos option.


For the details, here is the test code:

https://github.com/thin-edge/thin-edge.io/blob/635d25ccc1c00e4e18f98e35931d7a0fea24b63b/crates/core/tedge/tests/mqtt.rs#L32-L52

And we have system test for tedge mqtt pub. https://github.com/thin-edge/thin-edge.io/blob/main/tests/RobotFramework/tests/mqtt/basic_pub_sub.robot