[Bug] EKF2 Estimator Diverges After Brief Data Freezes - Need Help Diagnosing the Issue

[bynullname]

Describe the bug

Hello everyone,

I've encountered a perplexing issue when using a laser lidar for vision-based localization on my PX4 drone. I'm sending the laser lidar's TF data to the PX4 via the /mavros/vision_pose/pose topic and observing the EKF2 tracking in real-time through /mavros/local_position/pose. Everything seems fine initially, but about 1 minute into the operation, the estimator starts to diverge.

I've rigorously inspected the data I'm sending and even recorded rosbags to analyze the data quality, which appears to be quite good and consistent. Timestamps are also accurate.

What's really puzzling is that, when I looked at the ulg logs using PlutJugger, I noticed a freeze or disappearance in all flight controller data for 1-2 seconds just before the divergence. This issue also occurs when the drone is at rest, with SLAM and MAVROS turned off.

I've attached some data plots for further insights. Has anyone else encountered this issue or have any ideas on how to troubleshoot it?

Thank you!

To Reproduce

Setup: Initialize the PX4 drone and connect it to the computer running ROS and MAVROS.

Laser Lidar Data: Ensure the laser lidar is up and running. Send its TF data to PX4 via /mavros/vision_pose/pose.

Observation: Open a new terminal and use rostopic echo /mavros/local_position/pose to monitor the EKF2 tracking in real-time.

Wait: Keep the system running for approximately 1 minute.

Issue: Observe that the estimator starts to diverge.

Log Analysis: Download the ulg logs from the flight controller and use PlutJugger for analysis. Notice that all flight controller data freezes or disappears for 1-2 seconds just before divergence occurs.

Optional: Turn off SLAM and MAVROS and leave the drone at rest. Observe that the same issue of data freezing occurs.

Expected behavior

I expected the EKF2 estimator to maintain stable and accurate tracking of the drone's local position based on the high-quality and consistent laser lidar data being fed into /mavros/vision_pose/pose. In particular, I expected no interruptions or anomalies in the tracking process and certainly did not anticipate the estimator to diverge after a short period of operation.

Screenshot / Media

Yes,here is the /mavros/vision_pose/pose from slam data to mavros record by ros

Flight Log

https://logs.px4.io/plot_app?log=5cf34fa0-2d32-4668-9abf-8ad93a8a1406

Software Version

1.13.3 quadrotor x generic 250 racer

Flight controller

6c mini

Vehicle type

Multicopter

How are the different components wired up (including port information)

No response

Additional context

No response

[bynullname]

It seems that there is no problem with vision_pose, but local_position is wrong.

[dagar]

Could you give this a try with v1.14 or the main branch and I'll take a closer look at the logs? EKF2 external vision changed quite a bit between v1.13 and v1.14 and if there's any issue it'll have to be fixed in the newer implementation.

[bynullname]

I upgraded to 14, but unfortunately the data still breaks In the new version, I only modified the content of the log record, and the log recording method is as follows SDLOG_PROFILE 978 SDLOG_MODE from boot to shudown （because i need record data)

[bynullname]

Data stop time is up to 2 seconds

[bynullname]

Could you give this a try with v1.14 or the main branch and I'll take a closer look at the logs? EKF2 external vision changed quite a bit between v1.13 and v1.14 and if there's any issue it'll have to be fixed in the newer implementation.

ple see my update

[dagar]

Please share the log from v1.14. https://logs.px4.io/

[bynullname]

Please share the log from v1.14. https://logs.px4.io/

https://review.px4.io/plot_app?log=a2a69a40-ff7d-44fe-99cb-ede2c5710bfb This is the log of v1.14 I just plugged in the USB cable for power supply and kept it static for a while. When I read the Log, I will see interruptions in the data, but I can see it more clearly using plotjugger because it has a dotted plot.

Please share the log from v1.14. https://logs.px4.io/

[bynullname]

I think it’s not just a problem with vision fusion, but data will be interrupted. This interruption will inevitably cause fusion errors. data stop like this:

[dagar]

Those are logging dropouts, not the actual system freezing.

If you want to minimize them try switching to a much higher quality SD card, and you can also increase the logging buffer here. https://github.com/PX4/PX4-Autopilot/blob/7239e8473ed69d9a546d6e8e534b7b308b4a17e7/platforms/nuttx/init/stm32h7/rc.board_arch_defaults#L16

More realistically you need to reduce SDLOG_PROFILE. In particular the FIFO logging (Raw FIFO high-rate IMU) is quite expensive, I'd only turn those one when you need to review full 8 kHz raw accel/gyro. If you're still seeing dropouts after that try disabling "High rate".

I'd also keep the "Default set" enabled because it has a wide selection of topics at low rate.

[dagar]

@bynullname to be clear it's probably 2 separate issues 1) logging dropout 2) EKF fusion

First resolve the logging dropout, then get a clean log on v1.14 and we can review what's happening with the estimator.

[bynullname]

Thanks, After I selected only the default options for log, this problem no longer occurred.