[CI] Intermittent MAVROS mission test failures with segfaults

[supergra]

A number of recent CI MAVROS mission tests have been failing with segfaults.

I realized they were not deterministic failures by following the sequence on my PR:

https://github.com/PX4/Firmware/actions?query=workflow%3A%22MAVROS+Mission+Tests%22+branch%3Apr-fwatt-fixes

1) After two commits to the branch, the CI passed 2) After removing 3 PX4_WARN() statements, one MAVROS mission test failed (fixed wing). 3) After reverting that commit, a different MAVROS mission test (rover) failed, on the same state that it had passed in step (1) 4) After recommitting the 3 PX4_WARN() statements, all MAVROS mission tests passed, despite being the same state as (2).

I see these segfaults on some other recent CI runs:

• https://github.com/PX4/Firmware/runs/849945884
• https://github.com/PX4/Firmware/runs/844282215
• https://github.com/PX4/Firmware/runs/836591395

[julianoes]

The linked CI runs show SITL (MAVSDK) tests failing but not the MAVROS tests? Are you sure the MAVROS ones are actually failing?

[supergra]

I was just using the label of the CI test: "MAVROS Mission Tests". It is the SITL portion that is failing, yes. Feel free to change the title if it's misleading! I'm not familiar with MAVSDK or MAVROS, just saw that it couldn't possibly be a correct CI result.

[julianoes]

I see the following MAVROS mission test failures:

 INFO  [ecl/EKF] 5484000: starting GPS fusion
[ WARN] [1598377797.039020117, 18.388000000]: CMD: Unexpected command 176, result 0
INFO  [commander] Armed by external command
INFO  [navigator] Takeoff to 15.0 meters above home.
INFO  [commander] Takeoff detected
[ERROR] [1598377800.131651387, 21.480000000]: FCU: Critical failure detected: lockdown
ERROR [commander] Critical failure detected: lockdown
INFO  [commander] Disarmed by lockdown
INFO  [logger] closed logfile, bytes written: 2264884

https://github.com/PX4/Firmware/runs/1027845090#step:10:283

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.