flaky system test: tedge_connect_test_sm_services

[reubenmiller]

Describe the bug

Flaky system test.

Test name

tedge_connect_test_sm_services

Failures

message=Warning should not be displayed if the port does not match. Issue #2863: 'Checking if systemd is available.

Validating the bridge certificates.

Creating the device in Cumulocity cloud.

Deleting mosquitto bridge configuration in favour of built-in bridge

Restarting mosquitto service.

Awaiting mosquitto to start. This may take up to 5 seconds.

Enabling mosquitto service on reboots.

Successfully created bridge connection!

Checking if tedge-mapper is installed.

Starting tedge-mapper-c8y service.

Persisting tedge-mapper-c8y on reboot.

tedge-mapper-c8y service successfully started and enabled!

Sending packets to check connection. This may take up to 2 seconds.

Warning: Bridge has been configured, but Cumulocity connection check failed.

Enabling software management.

Checking if tedge-agent is installed.

Starting tedge-agent service.

Persisting tedge-agent on reboot.

tedge-agent service successfully started and enabled!

' contains 'Warning:'

Build seen on

• https://github.com/thin-edge/thin-edge.io/actions/runs/10147476910?pr=3020

To Reproduce

Invoked with:

invoke flake-finder --suite tedge_connect_test --iterations 5 --clean

Result:

Overall: FAILED
Results: 5 iterations, 2 passed, 3 failed
Elapsed time: 0:04:42.355737
Failed iterations: [1, 3, 4]

Expected behavior

The test should pass consistently.

[reubenmiller]

After some investigation it seem the failure is related to the built-in bridge functionality. All of the tests in the suite are executed on the same device, and the proceeding test inadvertently enables the built-in bridge when "cleaning up", see below for the test where this happens.

Non-root users should be able to read the mosquitto configuration files #2154
    [Tags]    \#2154
    Execute Command    sudo tedge connect c8y || true
    Should Have File Permissions    /etc/tedge/mosquitto-conf/tedge-mosquitto.conf    644 root:root
    Execute Command    sudo tedge config set mqtt.bridge.built_in false
    Execute Command    sudo tedge reconnect c8y
    Should Have File Permissions    /etc/tedge/mosquitto-conf/tedge-mosquitto.conf    644 root:root
    Should Have File Permissions    /etc/tedge/mosquitto-conf/c8y-bridge.conf    644 root:root
    Execute Command    sudo tedge config set mqtt.bridge.built_in true  # Reset things after running the test

As a first measure, the test should be updated to unset the mqtt.bridge.built_in setting instead of assuming it is of a particular value or not. The built-in bridge tests will be covered in another PR, https://github.com/thin-edge/thin-edge.io/pull/2911