MAVROS2: Failed to get parameter type

Vicidel commented 2 years ago

Issue details

I have a structure with multiple ROS2 nodes talking to each others. At boot, some of these nodes need to fetch PX4 parameters through MAVROS2. For some reason some params seem to always be fetched correctly, but some (for ex. GF_ACTION) seem to have issues most of the time, but it could well be complete coincidence...

Before migrating to MAVROS2 from MAVROS1 I never had the issue, so it might be something to do with this.

Mainly I have three nodes:

MAVROS2 node
safety node
- needs to get the GF_ACTION parameter before being ready
- does the param get with standard ROS2 interface /mavros/param/get_parameters
command node
- if it detects that other nodes (i.e. safety) has issues pulling parameters it sends a mavros_msgs::srv::ParamPull with force_pull = true to MAVROS node
- this was added to ensure that safety gets the params at some point, if I remove it I have cases where safety just never gets the GF_ACTION value I also have other ROS2 nodes in the setup, with three other nodes doing some param set in MAVROS2 at boot, but for the sake of clarity I'm not adding the logs there.

Thank you for any help of pointers to solve the issue.

MAVROS version and platform

Mavros: 2.1.1 ROS: Galactic (in Docker ros:galactic-ros-base-focal) Ubuntu: 20.04 (in Docker ros:galactic-ros-base-focal)

Autopilot type and version

[ ] ArduPilot [x] PX4

Version: v1.12.1

Node logs

I enabled debug logging for the MAVROS2 param node in order to get more information on what is going on behind the scenes. I'm attaching a small extract of my logs without the debug messages for clarity, as well as the complete mavros/command/safety nodes logs as files.

MT_04_0004_safety  | [1662357813.941736581] [INFO] Safety is getting params for drone MT_04_0004
MT_04_0004_mavros  | [1662357813.954224464] [WARN] PR: Failed to get parameter type: GF_ACTION
MT_04_0004_command | [1662357815.394246961] [INFO] PX4 param automatic pull triggered
MT_04_0004_mavros  | [1662357815.398018285] [INFO] PR: start force pull
MT_04_0004_safety  | [1662357818.941928040] [INFO] Safety is getting params for drone MT_04_0004
MT_04_0004_safety  | [1662357823.941805572] [INFO] Safety is getting params for drone MT_04_0004
MT_04_0004_safety  | [1662357823.944227132] [ERROR] Parameter GF_ACTION could not be obtained from MAVROS (timeout), will try again...
MT_04_0004_safety  | [1662357828.941820764] [INFO] Safety is getting params for drone MT_04_0004
MT_04_0004_safety  | [1662357828.947709830] [ERROR] Parameter GF_ACTION could not be obtained from MAVROS (timeout), will try again...
MT_04_0004_safety  | [1662357833.941750515] [INFO] Safety is getting params for drone MT_04_0004
MT_04_0004_safety  | [1662357833.943274406] [ERROR] Parameter GF_ACTION could not be obtained from MAVROS (timeout), will try again...
MT_04_0004_safety  | [1662357838.941787065] [INFO] Safety is getting params for drone MT_04_0004
MT_04_0004_safety  | [1662357838.942971437] [ERROR] Parameter GF_ACTION could not be obtained from MAVROS (timeout), will try again...
MT_04_0004_safety  | [1662357843.941778448] [INFO] Safety is getting params for drone MT_04_0004
MT_04_0004_safety  | [1662357843.943749580] [ERROR] Parameter GF_ACTION could not be obtained from MAVROS (timeout), will try again...
MT_04_0004_command | [1662357845.428300450] [ERROR] Could not pull params
MT_04_0004_mavros  | [1662357845.430796194] [WARN] PR: request param #61 timeout, retries left 2, and 797 params still missing
MT_04_0004_mavros  | [1662357845.444801878] [WARN] PR: Failed to get parameter type: GF_ACTION
MT_04_0004_command | [1662357845.448199511] [INFO] Detected MAVROS parameter issue, pulling params in 5s
MT_04_0004_mavros  | [1662357845.466269848] [WARN] PR: Failed to get parameter type: GF_ACTION
MT_04_0004_mavros  | [1662357845.492475064] [WARN] PR: Failed to get parameter type: GF_ACTION
MT_04_0004_mavros  | [1662357845.501926242] [WARN] PR: Failed to get parameter type: GF_ACTION
MT_04_0004_mavros  | [1662357845.509483437] [WARN] PR: Failed to get parameter type: GF_ACTION
MT_04_0004_mavros  | [1662357845.515120277] [WARN] PR: Failed to get parameter type: GF_ACTION
MT_04_0004_safety  | [1662357848.941784006] [INFO] Safety is getting params for drone MT_04_0004
MT_04_0004_mavros  | [1662357848.956923486] [WARN] PR: Failed to get parameter type: GF_ACTION
MT_04_0004_command | [1662357850.448727178] [INFO] PX4 param automatic pull triggered
MT_04_0004_mavros  | [1662357850.451589644] [INFO] PR: start force pull
MT_04_0004_safety  | [1662357853.942370151] [INFO] Safety is getting params for drone MT_04_0004

20220905_safety_param_pull.log 20220905_command_param_pull.log 20220905_mavros_param_pull.log

esharet commented 2 years ago

I have the same problem when trying to use service /mavros/param/get_parameters. I'm calling the service with call_async and I'm using mavros2, ros galactic and ardupilot (sitl).

Vicidel commented 2 years ago

While trying to create some reproducible code, I found out that running MAVROS2 alone with only one other node works fine. So the issue comes only when there are a lot of nodes doing requests to MAVROS2

I found a fix for my setup by doing a central param store on top of MAVROS2 for handling the params, like this it waits for the list from MAVROS2 and the other nodes requests from this store. It's a bit more complicated, but for my setup works fine.

vooon commented 2 years ago

@Vicidel cache likely be better than issuing tons of RPCs. Note that you can subscribe for parameter events and update cache sooner. https://github.com/mavlink/mavros/blob/904530dee98ead0799813ee2578f24bb48dc995f/mavros/src/plugins/param.cpp#L435-L438

mavlink / mavros