search

PX4 / PX4-Autopilot

PX4 Autopilot Software
https://px4.io
BSD 3-Clause "New" or "Revised" License
8.47k stars 13.5k forks source link

[Bug] Parameter download stops, reports unknown parameter IDs #21916

Open TorbjornHouge opened 1 year ago

TorbjornHouge commented 1 year ago

Describe the bug

Some times, not everytime and not in predictable circumstanses, parameter transfer seemingly stops. QGC only reports it failed, but looking in the log file from the AP, it list multiple errors of type "[mavlink] unknown parameter ID:". After multiple restarts, it suddenly starts functioning correct. The error only happens during initialization, no communications issues are experienced during flight.

To Reproduce

Not sure how to ensure reliable reproduction.

  1. Drone switched on
  2. Connect with QGC (either QGC running already or connecting after the drone has ran for some time both may lead to the same result)
  3. Wait for parameters to sync, if it fails QGC will report failure. If it doesn't fail this step takes 2-5 seconds

Expected behavior

When starting up the UAV and connecting, parameters should transfer and the system should initialize and get ready to fly.

Screenshot / Media

No response

Flight Log

https://review.px4.io/plot_app?log=9c34ff48-46d5-4104-b84d-7ca2e7bd1b8c

Software Version

PX4 1.13.3

Flight controller

Orange Cube

Vehicle type

Hybrid VTOL

How are the different components wired up (including port information)

The autopilot is on its custom carrier board that outputs TELEM1 to RS232. This is fed over to a Raspberry Pi 4 that uses Mavlink Router on port 14550 that QGC can connect to from the ground station.

Additional context

I am experiencing that the connection process with downloading parameters is highly inreliable. More often than not, it fails downloading parameters and I have to restart QGC or force a refresh multiple times before it succeeds (seems random to me). It happens both using UDP and TCP, possibly more often on UDP. However, TCP is not desirable once we start flying further away and get communication glitches. I tried Mission Planner which did not have any issues connecting and downloading parameters using UDP, which leads me to suspect QGC, but there may also be something else entirely. Connecting using wired Ethernet, the error happens less frequent, but it still happens. Connecting using RFD868EU telemetry radios it also occurs sporadically (however, due to the large number of parameters on a VTOL, I try to avoid using this due to its low bandwidth).

The error message is generated (note that this is where the log item is generated, it is not the cause of the error) here: https://github.com/PX4/PX4-Autopilot/blob/fc357921fd69e574a3f596be25faa8f4d9cdcea9/src/modules/mavlink/mavlink_parameters.cpp#L209

Also described here: https://discuss.px4.io/t/qgc-fails-downloading-parameters-upon-startup/32707/4

TorbjornHouge commented 1 year ago

We have now done some changes to the system, but most are the same as when the bug was reported. The problem still persist. I am wondering if we are doing something that triggers it to enable a lot more parameters than the "standard" setups. We use UAVCAN for actuators and sensors, and the system is a VTOL so there are a total of 10 actuators involved. Also interesting, I don't believe it is a datalink issue since today I was connected in the office. However it may have something todo with Mavlink-Router which is between the AP and QGC.

TorbjornHouge commented 1 year ago

I am still investigating this, not getting any closer to a solution. I am now observing the same both using RPi running mavlink-router, connecting directly to the serial port that the RPi is using (disconnecting the RPi in this case) and connecting to the Cube using USB. When connecting, the error rate jumps up to roughly 30%, then slowly sinks after this. This seems to lead to normal operating ok for most of the time, but the parameter download will fail more often than not. I have no idea how to get further with this, and I still suspect the root cause is that my combination of VTOL and UAVCAN actuators leads to the number of parameters getting too high for the system to reliably handle for some reason.

TorbjornHouge commented 1 year ago

After more checking and tweaking of parameters, the intermediate solution was to set MAV_0_RATE to 600 (from 1200). This is with SER_TEL1_BAUD set to 57600. It does look like it is working stable now, but it is strange that it should need to be that slow (it is around 1/4 of what the automatic setting will set it to).

TorbjornHouge commented 1 year ago

Problem is not completely solved by reducing MAV_0_RATE it seems, even though it is slightly better. It looks like it struggles with flight plans and no fly zones as well. One observation is that the fixed wing configuration has a parameter file with around 1000 values, while VTOL with Dronecan actuators has around 1300 values in parameters. Slight difference but should not be significant I think.