Incombatibility issues with PX4's most recent firmware

[may-ay]

Hello all,

When using PX4's most recent firmware (1.13.0, px4_fmu-v2_default.px4), I am unable to create a MAVSDK object.

If I update a PX4 (using PX4 mini at the moment) to the most recent version (1.13.0) using QGC, disconnect it from USB, plug it back in, and then attempt to create a MAVSDK object, it fails to create the object.

However, if I connect the same updated PX4, open up QGC, go through the "Downloading Parameters" portion, quit QGC, and then attempt to create a MAVSDK object, it will work.

Disconnecting the PX4 again will go back to the original issue of failing to create a MAVSDK object.

This issue occurs on MAVSDK 1.4.2 and 1.4.6.

If I revert my PX4 back to 1.12.0, I do not experience this issue.

Screenshot and example walk-through:

The first attempt was after updating the PX4 but the PX4 is still connected to QGC.

The second attempt was after updating the PX4, exiting out of QGC, but not unplugging the USB cable from the PX4.

The third and fourth attempt were after updating the PX4, exiting out of QGC, unplugging the USB cable from the PX4, and then connecting the USB cable back into the PX4.

[JonasVautherin]

Isn't it because over usb, PX4 now needs to receive the first MAVLink packets before it starts sending them? I know something like this has changed fairly recently, but I don't know exactly when.

Does this help? https://github.com/mavlink/MAVSDK/issues/1883

[may-ay]

Thanks Jonas, that was the issue.

Per the advice from #1883, adding the following lines to my code fixed the issue I was facing:

mavsdk::Mavsdk::Configuration configuration{
    mavsdk::Mavsdk::Configuration::UsageType::GroundStation};
configuration.set_always_send_heartbeats(true);

mavsdk::Mavsdk mavsdk;
mavsdk.set_configuration(configuration);

[mholzel]

Even after the proposed fix:

mavsdk::Mavsdk::Configuration configuration{
    mavsdk::Mavsdk::Configuration::UsageType::GroundStation};
configuration.set_always_send_heartbeats(true);

mavsdk::Mavsdk mavsdk;
mavsdk.set_configuration(configuration);

I am still seeing this issue on a Holybro Pixhawk 6x : https://holybro.com/products/pixhawk-6x?variant=42471703052477

I am trying to run the examples on commit v1.4.16.

I have tried flashing PX4 firmware 1.13.0 and 1.14.0beta through QGroundControl.

[dirksavage88]

Can confirm this is an issue over UDP with 1.13 as well. I am not sure what the issue is as I can get the new system info, an autopilot is added, and I see the partner IP in mavlink on the autopilot. The timeout occurs and I lose connection to the "ground station". I am using the offboard ports 14580 local and 14540 remote.

[julianoes]

@mholzel and @dirksavage88 please document the example you run and the full output, so that I get the full picture. Thanks.

[mholzel]

Here is a detailed list of commands to reproduce the issue:

First clone the repo and checkout the latest release:

matt@md:~$ git clone https://github.com/mavlink/MAVSDK.git
Cloning into 'MAVSDK'...
remote: Enumerating objects: 48757, done.
remote: Counting objects: 100% (1600/1600), done.
remote: Compressing objects: 100% (769/769), done.
remote: Total 48757 (delta 926), reused 1286 (delta 754), pack-reused 47157
Receiving objects: 100% (48757/48757), 32.60 MiB | 22.69 MiB/s, done.
Resolving deltas: 100% (31670/31670), done.
matt@md:~$ cd MAVSDK/
matt@md:~/MAVSDK$ git checkout v1.4.16 
Note: switching to 'v1.4.16'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at a14d604c Merge pull request #2073 from mavlink/pr-fix-rtk-python

All of the examples exhibit the same behavior, so I will focus on mavshell.cpp for now. In mavshell.cpp, I replace

mavsdk::Mavsdk mavsdk;

with

mavsdk::Mavsdk::Configuration configuration(
    mavsdk::Mavsdk::Configuration::UsageType::GroundStation);
configuration.set_always_send_heartbeats(true);
mavsdk::Mavsdk mavsdk;
mavsdk.set_configuration(configuration);

Now let's build it:

matt@md:~/MAVSDK$ cd examples/
matt@md:~/MAVSDK/examples$ mkdir build 
matt@md:~/MAVSDK/examples$ cd build/
matt@md:~/MAVSDK/examples/build$ cmake ..
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
------------------------------------------------------------
-- Configuring done (0.8s)
-- Generating done (0.0s)
-- Build files have been written to: /home/matt/MAVSDK/examples/build
matt@md:~/MAVSDK/examples/build$ make mavshell
[ 50%] Building CXX object mavshell/CMakeFiles/mavshell.dir/mavshell.cpp.o
[100%] Linking CXX executable mavshell
[100%] Built target mavshell

Next, we try running it. I have the Pixhawk 6x connected via a USB-C to USB-C cable to my laptop:

matt@md:~/MAVSDK/examples/build$ ./mavshell/mavshell serial:///dev/serial/by-id/usb-Auterion_PX4_FMU_v6X.x_0-if00
[09:11:25|Info ] MAVSDK version: v1.4.16 (mavsdk_impl.cpp:20)
Waiting to discover system...
No autopilot found.

I observe the same behavior on all of the examples... No autopilot found.

Let's try opening QGroundControl first:

matt@md:~/MAVSDK/examples/build$ ~/QGroundControl.AppImage 
Settings location "/home/matt/.config/QGroundControl.org/QGroundControl.ini" Is writable?: true
Filter rules "*Log.debug=false\nGStreamerAPILog.debug=true\nGStreamerAPILog.debug=true\nqt.qml.connections=false"
GStreamerAPILog: (NULL) Initializing GStreamer Core Library version 1.16.3
------------------------------------------------------------
GStreamerAPILog: <glcolorconvertelement1:src> Received event on flushing pad. Discarding
GStreamerAPILog: (NULL) trying to link sink:proxypad1 and gluploadelement1:sink
GStreamerAPILog: (NULL) linked sink:proxypad1 and gluploadelement1:sink, successful
GStreamerAPILog: (NULL) creating reconfigure event
GStreamerAPILog: <GstQgcVideoSinkBin@0x7fd5fc0101f0> adding pad 'sink'
"v4.2.0"

NOTE!!! At this point, the Pixhawk is not found by QGroundControl if the Pixhawk is plugged in before QGC starts. Therefore, while QGroundControl is running, I unplug the Pixhawk, and then replug in the USB-C connector on the Pixhawk. At this point, it is detected in QGroundControl, and I see additional output in the terminal (as well as the usual QGC GUI):

_recalcFlightPathSegments homePositionValid false
setCurrentPlanViewSeqNum
setCurrentPlanViewSeqNum
_recalcFlightPathSegments homePositionValid false
_recalcFlightPathSegments homePositionValid false
ParameterManagerLog: Attemping load from cache
ParameterManagerLog: Parameters cache match failed /home/matt/.config/QGroundControl.org/ParamCache/1_1.v2
QCoreApplication::postEvent: Unexpected null receiver
setCurrentPlanViewSeqNum
setCurrentPlanViewSeqNum
_recalcFlightPathSegments homePositionValid false
_recalcFlightPathSegments homePositionValid false
_recalcFlightPathSegments homePositionValid false

At this point it may be useful to note that if I go to the parameter section, and click "Show modified only", there are some parameters that are shown that I did not specifically modify. If I go to Tools->Reset all to firmware's default, then unplug and replug the Pixhawk, these changes persist:

Anyways... If I now close QGC by clicking the X in the upper right, then the example magically works:

matt@md:~/MAVSDK/examples/build$ ./mavshell/mavshell serial:///dev/serial/by-id/usb-Auterion_PX4_FMU_v6X.x_0-if00
[09:22:33|Info ] MAVSDK version: v1.4.16 (mavsdk_impl.cpp:20)
Waiting to discover system...
[09:22:33|Debug] New: System ID: 1 Comp ID: 1 (mavsdk_impl.cpp:496)
[09:22:33|Debug] Component Autopilot (1) added. (system_impl.cpp:377)
[09:22:33|Debug] Discovered 1 component(s) (system_impl.cpp:578)
Discovered autopilot

This specific example was run with the 1.13.3 firmware, but I have tried all of the built in options in QGC: stable, beta, master. I also compiled PX4 myself and uploaded the custom .px4 file, but that yields exactly the same result.

FWIW: here is what the Vehicle settings page looks like in the previous run

I have also tried calibrating the sensors.

It should also be noted the the shell in QGC seems to work normally. The sensor readout also looks excellent

As mentioned, I am trying to connect to the Pixhawk 6X over a USB-C/USB-C cable. I haven't gotten the ethernet connection to work yet.

I am on a one-year old Dell laptop running Ubuntu 20.04.6 with an Intel i9-12950HX cpu.

I installed MAVSDK using the .deb: libmavsdk-dev_1.4.16_ubuntu20.04_amd64.deb that I downloaded from here: https://github.com/mavlink/MAVSDK/releases/tag/v1.4.16.

I also sudo apt update on an almost weekly basis. There is nothing outdated there.

[mholzel]

One more important piece of info... I tried installing the Ardupilot firmware and running that, and it works without a hitch. However, the MAVSDK Telemetry data seems incomplete and low frequency when using Ardupilot. Hence I need to figure out the source of this issue with PX4.

[dirksavage88]

@julianoes this may be its own issue or related to @mholzel. I have been able to get mine to work on the offboard computer (Linux sbc) using the armv6 deb libmavsdk. However I see the no autopilot issue when connected via my development workstation (Ubuntu 20.04 Intel i7) regardless of if I have QGC connected or disconnected. I use the latest mavsdk and the lib for ubuntu amd64. Keep in mind I’m running mavlink offboard via this config: https://github.com/dirksavage88/Firmware/blob/2b27596cb93dbe1cf7558e35684c0842015ccfbd/posix-configs/bbblue/px4_offboard.config#L90

I will get an output later today. It’s probably something simple I’ve overlooked. But I can confirm I’m also running 1.13 and for my dev workstation the same mavsdk and libmavsdk mentioned above.

[julianoes]

@mholzel interesting, thanks for the detailed information. And let me try to reproduce this.

[julianoes]

However, the MAVSDK Telemetry data seems incomplete and low frequency when using Ardupilot.

For ArduPilot you have to manually request the telemetry streams at the rate you want them using the telemetry.set_rate_... functions.

[julianoes]

I reproduced this with a Pixhawk 6X and Pixhawk 6C, same MAVSDK release, on Ubuntu 22.04, and it works fine:

cmake --build build -j30 && build/mavshell/mavshell serial:///dev/serial/by-id/usb-Auterion_PX4_FMU_v6X.x_0-if00                      <<<
[  5%] Built target calibrate
[  9%] Built target follow_me
[ 12%] Built target autopilot_server
[ 20%] Built target battery
[ 20%] Built target camera
[ 24%] Built target fly_mission
[ 27%] Built target camera_settings
[ 35%] Built target fly_multiple_drones
[ 38%] Built target ftp_server
[ 38%] Built target ftp_client
[ 42%] Built target fly_qgc_mission
[ 46%] Built target geofence_inclusion
[ 50%] Built target gimbal
[ 53%] Built target terminate
[ 57%] Built target offboard
[ 61%] Built target multiple_drones
[ 64%] Built target gimbal_device_tester
[ 70%] Built target set_actuator
[ 83%] Built target takeoff_and_land
[ 83%] Built target logfile_download
[ 83%] Built target tune
[ 85%] Built target manual_control
[ 88%] Built target vtol_transition
[ 92%] Built target parachute
[ 96%] Built target mavshell
[100%] Built target transponder
[02:17:55|Info ] MAVSDK version: v1.4.16 (mavsdk_impl.cpp:20)
Waiting to discover system...
[02:17:56|Debug] New: System ID: 1 Comp ID: 1 (mavsdk_impl.cpp:496)
[02:17:56|Debug] Component Autopilot (1) added. (system_impl.cpp:377)
[02:17:57|Debug] MAVLink: critical: Preflight Fail: ekf2 missing data (system_impl.cpp:250)
[02:17:57|Warn ] Vehicle type changed (new type: 2, old type: 0) (system_impl.cpp:225)
[02:17:57|Debug] Discovered 1 component(s) (system_impl.cpp:578)
Discovered autopilot
[02:17:57|Debug] MAVLink: critical: Preflight Fail: Compass Sensor 0 missing (system_impl.cpp:250)
[02:17:57|Debug] MAVLink: info: Preflight Fail: No manual control input  (system_impl.cpp:250)

nsh: sysinit: fopen failed: No such file or directory

NuttShell (NSH) NuttX-11.0.0
nsh>

Also, the heartbeat config change is not required since https://github.com/mavlink/MAVSDK/releases/tag/v1.4.11.

I'm not sure what is messing with your serial access. :thinking:

[mholzel]

Is there something I need to modify on the pixhawk SD card? Some parameter or config?

I can test a different laptop with Ubuntu 22 today.

Also, is there a way to generate a more verbose log to maybe understand why the serial connection is failing?

[mholzel]

I tried with a laptop running Ubuntu 22 using the .deb: libmavsdk-dev_1.4.16_ubuntu20.04_amd64.deb.

It worked out of the box. So now I need to debug what is going on with the serial connection on the other laptop.

[mholzel]

So something strange is going on on one of my laptops (the original one that I noticed the problem on).

I spent several hours modifying the udev settings, trying several cables (USB A-C, USB C-C), trying several USB ports on my computer, rebooting, trying bash without modifications (bash --noprofile --norc), and I think that I am narrowing in on a culprit...

There seems to be a very narrow window, where I can get the example to work. Specifically, if I connect the Pixhawk, and then see the Green PWR indicator on the ethernet side, then I have about literally 1 to 3 seconds to start the example, and it will work. Too soon or too late, and it will not work. My guess is that on the new laptop where I am seeing the issue, that if the laptop doesn't get a signal from the Pixhawk within some specific period, then it is putting that USB port to sleep or something. And that time is probably a few seconds longer than the Pixhawk boot sequence.

I don't know though. That's my best guess for now.

On the old laptop, I don't see this behavior.

[mholzel]

Ah yes. I think that that was the issue...

It appears that the new laptop I have is putting the USB ports to sleep faster than my other machines (or not putting them to sleep at all). This is strange because the USB port is clearly still getting power, and clearly you can "connect" to the Pixhawk because there is a tty connection available, but somehow the link is down.

The fix was simply this: Modify grub to not put the USB ports to sleep.

Specifically, this file:

/etc/default/grub

I modified the line:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

to read:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash usbcore.autosuspend=-1"

Then you

sudo update-grub
sudo reboot

and viola...

[julianoes]

Oh wow, tricky! Nice detective work and thanks for following up and providing a workaround!

[mholzel]

@julianoes I did notice one strange thing when debugging the udev rules... some places in the docs mention the pixhawk vendor ID as 26ac (if memory serves me right), but when I was debugging the USB logs, I saw 3185. Do you know why there is a discrepancy?

Also, if memory serves me right, the Ardupilot examples would crash when I tried to set the rate. Maybe I can reflash the Ardupilot firmware tomorrow to double check, but it wasn't working out the box. I believe I calibrated the sensors in QGC.

[mholzel]

For completeness, I tried putting Ardupilot on the Pixhawk, and calibrating all of the sensors. Here is the QGC summary:

When I try to log the imu, I cannot set the rate. Specifically, this code:

    auto telemetry = Telemetry{system};
    const auto set_rate_result = telemetry.set_rate_imu(10);
    if (set_rate_result != Telemetry::Result::Success) {
        std::cerr << "Setting rate failed: " << set_rate_result << '\n';
        return 1;
    }
    telemetry.subscribe_imu([](mavsdk::Telemetry::Imu imu) {
        std::cout << "imu: " << imu.acceleration_frd.forward_m_s2 << " acceleration_frd.forward_m_s2" << std::endl;
    });

produces output:

[09:15:15|Info ] MAVSDK version: v1.4.16 (mavsdk_impl.cpp:20)
Waiting to discover system...
[09:15:15|Debug] New: System ID: 1 Comp ID: 1 (mavsdk_impl.cpp:496)
[09:15:15|Debug] Component Autopilot (1) added. (system_impl.cpp:377)
[09:15:15|Warn ] Vehicle type changed (new type: 2, old type: 0) (system_impl.cpp:225)
[09:15:15|Debug] Discovered 1 component(s) (system_impl.cpp:578)
Discovered autopilot
[09:15:15|Debug] MAVLink: info: No ap_message for mavlink id (105) (system_impl.cpp:250)
Setting rate failed: Command Denied

[julianoes]

So the message requested is HIGHRES_IMU, which is 105, and that's answered with No ap_message for mavlink id (105) here: https://github.com/ArduPilot/ardupilot/blob/36ffe42f01b3789d6351039264405d936f18b5bd/libraries/GCS_MAVLink/GCS_Common.cpp#L2831

So it looks like HIGHRES_IMU is not in the list supported to get at a certain rate? I wouldn't know...