Too many false positive "Preflight Fail: HIGH IMU ACCEL BIAS" warning messages

[Antiheavy]

Problem:

We are getting a lot of false positive "Preflight Fail: HIGH IMU ACCEL BIAS" warning messages preventing arming. This seems to have started in master this past spring and continues in stable 1.8.0. I believe a lot of other PX4 users are getting these false positive warnings as well.

• (Note: in recent master the text of this warning as changed to "Preflight Fail: High Accelerometer Bias")

This happens to us frequently on known good aircraft that are great fliers and continue to fly great once the fault is cleared.

We recently noticed the error seems correlated to a vehicle sitting level on the ground for some period of time. @priseborough made a helpful suggestion to try pitching and rolling the vehicle slowly so that the X and Y accels could spend some time facing down (gravity) and EKF2 could better estimate their bias. This worked to clear the error! Here is an example of the error which was then easily cleared by pitching and rolling the vehicle: https://review.px4.io/plot_app?log=278fa376-2a02-4027-b465-12507dbc9062

Possible solution: It has been suggested that using the bias state uncertainty as a factor in the bias level check could help eliminate false positives. The bias level check in Commander could use the already existing additional information from EKF2 for this.

Further support:

Anecdotally I know of this exact ACCEL BIAS warning happening to handfuls of other PX4 users, many of whom do not post to Github or Discuss.

There are a large number of github issues and PX4Discuss posts about getting this error. Probably there are many complicating factors involved with each report, but the fact that there are so many makes me believe there is a systemic issue with the check, or the checks and warnings need to be more specific. https://github.com/PX4/Firmware/issues/9241 https://github.com/PX4/Firmware/issues/9474 http://discuss.px4.io/t/preflight-fail/7023/10 http://discuss.px4.io/t/ekf2-problem-z-velocity-estimate-weird-behaviour/8395 http://discuss.px4.io/t/auto-land-ekf2-test-ratios-flight-controller-reset/8264

[Antiheavy]

FYI @priseborough @dagar

[priseborough]

@Antiheavy What is the cause of the fluctuating IMU readings immediately after startup?, eg

Is it a motion sequence that you have found is able to trigger this fault?

[Antiheavy]

I spoke with the pilot of this log and he said this motion is typical ground handling of this hand launched, fixed wing, foam aircraft. It was windy that day and the vehicle was getting buffeted around in his hands while he walked out to the takeoff and landing location. The vehicle was powered on and booted up while the pilot was walking with it. This type of ground handling is typical for hand launched vehicles.

We will try to see if we can make a repeatable method to get the accel bias error. So far it seems random, but more testing is worth while.

[Antiheavy]

Is it a motion sequence that you have found is able to trigger this fault?

we've tried a bunch of times to see if this motion sequence on boot-up was able to reproduce this. We also tried temperature changes after boot (move vehicle from warm inside to cold outside). We have been unable to reproduce it reliably.

However, it still happens seemingly randomly.

[JuanCruzAyoroa]

I'm having the same problem on Pixhawk 2.1, PX4 V1.8.1. I'll try the manual pitching/rolling workaround next time. Some questions:

1) Could this be a safety issue? Or it should just be an annoyance? 2) Is there any update on a more permanent fix?
3) Do we know at which release it started? I might consider using an older release.

Thanks!

[priseborough]

Reverting is unnecessary. As a short term fix until the check is improved to eliminate false positives, increasing the value of COM_ARM_EKF_AB will make it go away.

[Antiheavy]

@priseborough is there any additional testing you can recommend that will help with implementing the improved check? It has proven difficult to reproduce reliably.

[JuanCruzAyoroa]

Thanks, I'll increase the COM_ARM_EKF_AB for now.

I haven't been able to find a pattern that produces the error, that would help us debug it. I'll report back if I get new insights.

[Kjkinney]

https://review.px4.io/plot_app?log=f176556e-9a43-40c4-bbc7-e0e65895ec7d

https://review.px4.io/plot_app?log=9e0f5243-7cce-4db6-827a-f720abda7d64

Here are two logs with accel bias errors prior to flight. For both of these logs I restarted the vehicle in order to clear the errors.

Prior to these errors the vehicle did go through substantial temperature changes. The outside temperature was around 18F. For the first error the vehicle was sitting inside before being put outside and immediately powered. The second Occurred after the vehicle was sitting outside for about 15 minutes and then powered up.

[jzazbert]

Not sure if this is related, but I was having a bunch if ACCEL inconsistency issues with PIxhawk 2.1 cubes. I added some debug messages into my firmware and discovered that one of the accels had a lot of change in bias over temperature. We started seeing these errors when the weather turned cold.

I was above to show very large biases on one of the accels up to 1m/s by chilling the aircraft in a thermal chamber to -20C.

I ended up doing the recommended temperature calibration over -30C to 65C in a thermal chamber, and my issues went away. Reference here: https://docs.px4.io/en/advanced_config/sensor_thermal_calibration.html

My issue was specifically the inconsistency among the 3 accels on the cube, but if your accel bias was varying as much as mine with temperature, it could have triggered this error as the system warmed up.

[qhmb]

I have the same problem with the "Preflight Fail: HIGH IMU ACCEL BIAS". But after the restart of the drone, the same error appears. I have to recalibrate the ACC with QGroundcontrol to get the drone fly again. Then the drone flies about 2 times and afterwards the error appears again. I have multiple drones with the same HW and SW configuration. One of the drones does not show the error during all flights. But the others always show the "Preflight Fail: HIGH IMU ACCEL BIAS" error.

I'm using the EKF2 with Vision for indoor navigation. Is there a way to check if the "Preflight Fail: HIGH IMU ACCEL BIAS" error appears due to the bug?

[pietrodn]

I have the same problem (intermittently) when flying indoor with motion capture. I have partially coped by setting the COM_ARM_EKF_AB parameter higher (around 3.5e-3), but that's not a solution.

[LorenzMeier]

@priseborough Could you have a very brief and quick look at more of these instances and see if we're too specific on the warning or if we need better detection?

[TSC21]

I confirmed this issue with people trying to fly indoors as well.

[priseborough]

I will have a quick look through the logs tomorrow. Previoyus logs inspected have indicated that it was partially due to initial larger fluctuations in the X and Y (perpendicular to gravity vector) delta velocity bias errors associated with sensor errors and poor observability prior to flight because velocity change or gravity vector are required to make biases observable. The fact that these errors have been reported as being able to be cleared by tilting the vehicle to align X and Y in turn with the gravity vector supports this hypothesis.

@TSC21 Can you confirm if rotating the vehicle to temporarily align the X and then Y axis with the gravity vector clears the error?

A potential solution I discussed previously with the OP was to use the IMU delta velocity bias state variances to scale the check threshold in the commander so that checks are relaxed until the states become more observable.

[TSC21]

@TSC21 Can you confirm if rotating the vehicle to temporarily align the X and then Y axis with the gravity vector clears the error?

I will check and let you know.

[shrit]

I am getting this error during the Gazebo SITL. I am using IRIS quadcopters without optical flow. I reset the quadcopters using a specifc model for gazebo. However, after resetting the quad I am getting this error and I was not able to clean the error

[Antiheavy]

a related issue post about this topic: https://github.com/PX4/Firmware/issues/9790

[Antiheavy]

We went test flying yesterday and had 6 different vehicles all showing Accel Bias issues. It was pretty cold outside (-9 C) so we are thinking temp drift is the main cause. here is an example log from one of the six vehicles:

https://review.px4.io/plot_app?log=caf7018d-23d1-4007-bc28-54c078007b20

[priseborough]

If affected users are able to try https://github.com/PX4/Firmware/pull/11350 and report back, then that will help with testing. TXS.

[thomasgubler]

Closed via https://github.com/PX4/Firmware/pull/11350

[AlexWUrobot]

There is a similar issue happening in the simulation environment. https://discuss.px4.io/t/simulation-take-off-motion-not-vertical/30705

[bresch]

@AlexWUrobot Please open a new issue, this was another issue encountered 5 years ago...