PX4 SITL Tests Improvements

[mrpollo]

We need to improve the current SITL tests.

Stage 1

• [ ] Improve specific failing tests
  • [ ] (vtol) Fly VTOL mission with wind change (ramped_up_wind.world)
  • https://github.com/PX4/PX4-Autopilot/actions/runs/11940615234/job/33283663787#step:19:3117
  • https://github.com/PX4/PX4-Autopilot/actions/runs/11946116827/job/33299931473#step:19:3089
  • [ ] (vtol) RTL direct home with approaches
  • https://github.com/PX4/PX4-Autopilot/actions/runs/11941237714/job/33285631449#step:19:2169
  • https://github.com/PX4/PX4-Autopilot/actions/runs/11943817019/job/33293577322#step:19:2172
  • https://github.com/PX4/PX4-Autopilot/actions/runs/11963888081/job/33355197712?pr=24020#step:19:2401
  • [ ] (vtol) RTL direct home not as rally point
  • https://github.com/PX4/PX4-Autopilot/actions/runs/11904739730/job/33174054458?pr=23958#step:19:2461
  • [ ] (VTOL) RTL direct Mission Landing
  • https://github.com/PX4/PX4-Autopilot/actions/runs/11980160836/job/33403865204#step:19:4046
  • [ ] (trailsitter) RTL direct Mission Land
  • https://github.com/PX4/PX4-Autopilot/actions/runs/11941448644/job/33286304485#step:19:1446
  • [ ] (tailsitter) RTL direct rally without approaches
  • https://github.com/PX4/PX4-Autopilot/actions/runs/11944522384/job/33295599685#step:19:2865
  • [ ] (tailsitter) RTL direct rally with approaches
  • https://github.com/PX4/PX4-Autopilot/actions/runs/11950993944/job/33313676674?pr=24011#step:19:3231
  • https://github.com/PX4/PX4-Autopilot/actions/runs/12006914556/job/33466430076?pr=23961#step:19:3272

Stage 2

• Update architecture to use the latest dependencies, Base OS, PX4, and MAVSDK
• Migrate to the latest Gazebo (not classic)
• Document how SITL tests work in Github Actions

[MaEtUgR]

Can we coordinate on fixing the tests instead of just disabling them. I'm pretty sure they point us to real issues and we just gloss over because it's too hard to figure out what's going on in an hour.

I see two failure cases:

1. VTOL related to airspeed validation dropping out

   • Airspeed module should not be down in the first place not at the beginning and not mid flight, airspeed data is available the whole time.
   • Transition fails because the check considers outdated airspeed: https://github.com/PX4/PX4-Autopilot/blob/2f65644aeb5cb4c2023e43ac45d8a9b01d0584ba/src/modules/vtol_att_control/vtol_type.cpp#L181 I quickly discussed with @sfuhrer

2. Multicopter almost any test type can time out without arming because "ekf2 missing data"

   • It's running multi EKF ~and the failing test at least had instance_changed_count 1~ passing tests as well.
   • Blindly increasing timeouts like in https://github.com/PX4/PX4-Autopilot/pull/23892 doesn't help because the state is screwed.

[mrpollo]

Yes, @MaEtUgR, we are changing gears here and started working on fixing the tests after we fixed the noisy issues that were hiding the real problems

[dagar]

This one looks real, during RTL there's a bogus "Pilot took over using sticks" and we have a fly away.

standard_vtol: 'RTL with Mission Landing': failed https://github.com/PX4/PX4-Autopilot/actions/runs/11957851318/job/33335973614

https://logs.px4.io/plot_app?log=e0673b3c-8514-4390-b3ee-d00a317b17e3

[MaEtUgR]

A shot in the dark. Probably not the issue but might give us more insights also if it only rules out multi-EKF: https://github.com/PX4/PX4-Autopilot/pull/24017

[MaEtUgR]

@dagar Regarding pilot took over I'm confused. Plotjuggler doesn't show this message (event?). As expected there's no valid stick input and also never a sticks moving flag. The mode never changes to Position or indicates switching to Position/Altitude is not possible but that would be what "Pilot took over" stands for.

EDIT: Manual control is neither updated nor valid, nor moving. So I'd put my bet on the message being mixed up 🤔 https://github.com/PX4/PX4-Autopilot/blob/2f65644aeb5cb4c2023e43ac45d8a9b01d0584ba/src/modules/commander/Commander.cpp#L2925-L2933