Plane: Detect bad airspeed readings

[Naterater]

Feature request

Is your feature request related to a problem? Please describe. Partial airspeed sensor failure can be very dangerous for planes that cannot structurally handle higher speeds. Dirt or disconnected/broken tubes can cause false low airspeed readings that are not detected by AHRS as high wind. No error is detected until it is too late and the plane exceeds its maximum structural airspeed.

AHRS_WIND_MAX does NOT detect this failure. The wind estimate oscillates and does not climb if you're flying in multiple directions; it doesn't climb to the threshold.

Describe the solution you'd like 1: Detect the failure, display it as a message "potential airspeed calibration error." Common indications of airspeed calibration failure would be wind shifts >90 degrees in <60 seconds and >5m/s change. GPS speed consistently GREATER than airspeed for ground tracks or flight lines in opposite directions. 2: Give a failsafe option that is configurable. Primary response would be to enter throttle mode -- ARSPD_USE = 0. Other mode would be to use the autocal ARSPD_RATIO from the automatic calibration code 3: Return to home or continue based on configuration.

Describe alternatives you've considered Dual airspeed sensors

Platform [ ] All [ ] AntennaTracker [ ] Copter [x] Plane [ ] Rover [ ] Submarine

Additional context My planes enter uncommanded death rolls when flown too fast. After the dive starts, TECS never lets off of the throttle because of the altitude being too low. Pitch control is effectively lost as well.

[ghost]

Additional option: using 2 airspeed sensors (either on i2c or one analog other to i2c)

[Kelly-Foster]

+1 on Nate's request. I had a broken pitot tube (outer tube not well connected to inner tube) and the result was that RTL tried hard fast dives that would have ended in a smokin' hole had I not switched to Manual in time. Once Tridge diagnosed that bad airspeed values were the root cause, I wondered if there wasn't some combination of other logic that could conclude that the hard fast dive maybe wasn't the right flight plan.

Kelly

[Naterater]

This might even solve some EKF issues. Could the airspeed_autocal code be used?

[Naterater]

This issues has just popped up again for me and for another person with a very lucky save (facebook ardupilot.org post: https://www.facebook.com/steffen.reuter.33/videos/10211248880491619/)

My issue - hole on airspeed line during takeoff. Caused PIDs to be unstable immediately on launch - just about burnt out servos in Auto and FBWA modes from the rapid oscillations. 100% throttle until recovery. No warnings on GCS. True airspeed - approx 30m/s and indicated 7m/s.

Issue on FPV plane - during RTH, airspeed sensor failed - possibly ice? - and caused unstable PIDs and plane nearly hitting the ground. No logs to show, but video is clear on what happened. User recovered in manual mode.

[proficnc]

This is a very important issue, and one that I have experienced myself.

I would love for a first step, the ability to have two sensors or more, to either remove an outlier if we had 3 or more sensors, or to disable the scaling and cause a failsafes if the two disagreed with each other by too much.

Maybe some other options would also be viable

@bugobliterator can you please add this to our January discussion list?

[pompecukor]

I (my clients) have lost UAVs doe to this. IT flew into rain and pitot got blocked by moisture. In my opinion does As would just add complexity. I am sure there is a way to compensate for this in the code. The think that we all know happens is that the plane switched to max throttle, which in itself would not cause a crash, the bigger issues is that it starts to (force) pitch down which almost always ends in a crash. Most of these planes can fly very well with throttle mode only, just sacrifice a bit of efficiency. I think most of us would sacrifice that bit of efficiency any day for just to know that the plane will still get home. So even if the feature is "over reactive" it will not be a problem. That is there is an acceptable level of false positives.

[proficnc]

Flying an aircraft into rain that is not designed to fly into rain is a critical flaw.

The cheap pitot tubes available online are only designed for dry non icing conditions.

If you wish to fly in wet or icing conditions, then you must use an pitot/static probe that is designed for such conditions.

That is not a code issue

[pompecukor]

ProfiCNC I just posted this on FB, so I will just quote:

It is not as simple as this. Try to look at the bigger picture. I, just like Nate has had customer lose a plane to exactly this. Simple mapping mission on a normal day and suddenly is started raining 3km away from base. There is little to no weather forecast in Africa. Even if you have it is never reliable. So such will happen. The plane would 99% fly in rain except for the pitot. Which gets blocked (As it did in this instance too) and the plane would do max throttle and dive into the ground. So it is not by design or expertise. Sometime it is just bad luck. I too have considered not using AS at all as the planes fly very well even without that. A tad bit less efficencent, but FW had too much endurance these days anyway ;). However most of my users are not pilots and rely on 100% autonomous landing. With reversing throttle. Hence AS is a must.

[proficnc]

As I answered. You need to design or purchase an airspeed sensor that is fit for purpose at this point.

Designing a detector for a single faulty sensor is a very good thing, and I obviously fully support that, but this is also something that requires mechanical engineering to fully solve.

You can do that bit now, without waiting for anyone to code anything.

[Naterater]

This has raised in numerous issues since 2014 - all closed. It has never been fixed except by adding another sensor... that still doesn't do the job correctly in a few cases like a blocked pitot tube.

https://github.com/ArduPilot/ardupilot/issues/1215 https://github.com/ArduPilot/ardupilot/issues/1628 https://github.com/ArduPilot/ardupilot/issues/2787 https://github.com/ArduPilot/ardupilot/issues/3159 https://github.com/ArduPilot/ardupilot/issues/3188 https://github.com/ArduPilot/ardupilot/issues/5951 https://github.com/ArduPilot/ardupilot/pull/4122 https://github.com/ArduPilot/ardupilot/pull/7738

The EKF is intelligent at removing data with errors, yet it fails at detecting bad airspeed. AHRS_WIND_MAX does not protect you (even though the parameter description says it will), and the current code behavior is dangerous in my opinion. There are quite a few

There are certain things that need to be prioritized, and this is one of them.

[pompecukor]

AHRS_WIND_MAX does not protect you (even though the parameter description says it will)

Yes, that is something I had to find out the hard way. Based on the description, we all made the assumption that if you set AHRS_WIND_MAX to 10m/s for example then if for example your AS gets blocked by rain drop or just gets unplugged or what every else and hence start to report zero (or close to zero). and ground speed is reading more than 10m/s. Like 15m/s, which is way over. Then the code would fall back to cruise throttle and stop using AS (pretending the AS does not even exist). Hence would fly home and maybe have a uglier landing than usual but not fly into the ground at 20 degrees pitch down and 36m/s.

I do not code, and am not counting on freebies. Not demanding anything either. I get way way more from autopilot org that I can ever give in return, even tough we do what we can from our part with, testing bleeding edge stuff with expensive equipment and report what we find, suggesting features, give general feedback and last but not least, donating to the organization.

That being said. With my limited knowledge of computer programming. I quite sure that adding such fall back (that is to make AHRS_WIND_MAX to work as described) is not rocket science for the awesome team working on the code. I am almost sure that it was an oversight in the code. Otherwise why would the description say different.

For those that are worried about using it, there is "0" to disable. But for all the rest this would be a life (plane) saver. Maybe it is my limited understanding, but I do not see how fixing this now would require anything to do with mechanical engineering. @proficnc ?

Yes, there can be very in-depth solutions down the road. However for now, just a basic "correction" like this will already go a long way.

I am sure anyone sane would chose a plane to switch to "throttle mode" over "dive of death" when faced with such choice. Those of us that are well versed in flying can of course take over (Manual or FBWA) as we mostly do if we need to. But for users who cannot fly and/or cannot react fast, then throttle mode is always going to be the better choice.

[WickedShell]

I'd welcome improvements in this area. I have a patch around here somewhere that if your airspeed innovations in the EKF are to high for a long enough period of time would turn off using the airspeed sensor. Unfortunately there where 2 problems with:

• Only worked on very large step changes (IE instantaneous change of 100 pascals), otherwise the filter in the EKF drifts with the bad sensor.
• Requires a fairly long period otherwise momentary upsets have a much to high false positive chance, and required the airspeed to be appropriately calibrated/installed.

It's really the former one which was an issue, any somewhat slow change gets incorporated into the EKF very quickly which causes problems with doing the reset.

With all that being said if you can share them please post logs of airspeed failure. I only have a couple of my own logs for looking at the EKF innovations on this, and couldn't get to any solution which worked well/covered any of the actual failure logs I have. (They are either slow failures, or one with the pitot tube flopping in the wind, but even on that one as it flops through the correct airspeed the EKF innovation lowered which resets the timer).

[pompecukor]

@WickedShell , I am happy to share logs with you, but can only do it privately as it from a client. That is exactly the one from the picture that I posted above, which got into sudden heavy rain. Maybe Nate has some that he does not mind sharing too.

AirSpeed started to oscillate between 0 (in which case it went full throttle and pitch down) to normal airspeed (in which case it went back to normal flight). The oscillation was very slow, like 30 seconds interval. I am guessing, sometime the moisture was able to get out of the tube, but then after another 30 seconds if hit another one. So it ends up looking like a roller coaster. Let me know.

[ChrisBird]

I'm going to try and fix the AHRS_MAX_SPEED method to see if that provides an effective way of dealing with it. I'll have to see what else is currently using its logic and ensure I dont mess anything else in the change. It seems to be the simplest option and can be turned off by setting it back to zero anyway.

Should be a good one to sink my teeth into while I wait for Black Magic to send me a demo camera to work on the integration since my BM Pocket CC 4k is delayed.

[ChrisBird]

I think I've got the wind_max working as we'd expect it to. I'll do a code tidy, some further testing and then if it looks good I'll create the PR tomorrow night. This should stop people cratering their planes if all goes well.

I've got some delays built in so that we dont turn off the airspeed sensor use if it's a handful of bad readings. Whats a reasonable amount of time people would like to say 'hmmm, its x amount of time, this thing is not going to get better?' I've currently got 1 second but it might be too short..... Thinking it should be a few seconds just in case its due to some other valid issue.

[pompecukor]

@ChrisBird much appreciated man!!!

Good question about delay. I think it should be a few second. 4 maybe. Definitely not less. Maybe can be a bit more. Also I wonder. Just is have fall back, if for example the dew drop dries our?

Thanks.

[ChrisBird]

No probs @pompecukor. I dont think its the complete solution but its a step in the right direction. No fail back other than manually updating arspd_use to 1 again.

[auturgy]

Scaling the timeout with speed might be an idea. One way to avoid the ekf filter issue would be to use an independent synthetic airspeed estimate from within the airspeed library. An interesting approach is here: http://gallinazo.flightgear.org/uas/synthetic-air-data-an-afternoon-hack/

[ChrisBird]

I've created the PR, I'll be making further changes to enhance it (such as resetting the counter after an amount of time of good readings). Not a silver bullet for this issue but at least AHRS_WIND_MAX's documentation will be correct :-)

[ChrisBird]

I've reworked the PR. It is a result of the dev call discussions. Its now all handled within the AP_ARSPD library. It still uses the parameter AHRS_WIND_MAX.

Its now an option that is configurable by a new ARSPD_OPTIONS bitmask.

It works on each individual sensor, it checks for 5 seconds worth of bad readings over 60 seconds. If it meets that threshold then it sets that airspeed sensors use to 0. Once that happens it will then require 30 seconds of good readings over 60 seconds to enable the sensor use again. That has smoothed out the enabling.

We can rejig those figures if needed.

With some luck this one will make it.

[ChrisBird]

There were a number of comments which have proved to be very useful and improve the efficiency of the implementation. So basically I've reworked it to use a low pass filter on the health of the sensor. It has a rapid fall in health when its bad and a slow rise when its healthy.

I've added a warning level too with this, so if you happen to be right at the limits of the wind and gusts are nudging it then when the confidence level lowers it will warn the user via the gcs that its hit a warning condition.

It will only do this basic check if its enabled by the ARSPD_OPTIONS bitmask. The first bit enables this check, the second will enable the reenabling of the sensor when it's got a good confidence level.

It takes about 4 to 5 seconds to trip a failure from known OK confidence level for a complete 100% bad readings. Takes about 45 to 60 seconds to be enabled again if close to 100% good readings from a really low confidence level.

We are getting closer to a solution, hopefully this will get across the line soon.

[rsflylogix]

Hi @ChrisBird,

I was wondering if the failure check is already part of the code base?

[acobo]

Hi @ChrisBird, I was wondering if the failure check is already part of the code base?

Hi @rsflylogix , I think that the change was commited to master last 3-feb-2019: https://github.com/ArduPilot/ardupilot/pull/10243/commits/d3fbe386f3075ddde09d1a3f341a199464c69dd2 But it is not in the current stable version, Plane 3.9.6

[ChrisBird]

Hi @rsflylogix,

Sorry for the delay, was away for the Dev conference and when I got back I had a few tasks to catch up on.

Its in master it will take the next minor release to plane for it to go in. So that would likely be 3.10, when that will happen is anyone's guess. Plane (Tridge) seems to be very good at kicking out new version, so it'll likely be no more than a few months.

If you need a copy of Plane 3.9.6 with it let me know and I can build you one. Note that the custom build wont have gone through the normal beta testing though (shouldnt be an issue as it's been tested to death in SITL and one of my planes). Or you can build your own, just take the Plane 3.9.6 branch and apply the commits to it.

Chris

[IamPete1]

We now have dedicated warning and disabling wind speed thresholds,

https://ardupilot.org/plane/docs/airspeed.html#failure

and also EKF3 lane affinity.

https://ardupilot.org/plane/docs/common-ek3-affinity-lane-switching.html#common-ek3-affinity-lane-switching

So I would like to close.

[ddomit]

Can someone help me figure out what happened on this flight? It was supposed to do qrtl but instead it switched to RTL with a extremely large radius, about 1km. Airspeed was extremely high. My guess is that its something to do with the airspeed sensor but i want to understand:

• Why did it changed qrtl to RTL and why the RTL radius was so high.
• What could have been done to prevent this?

This unfortunately resulted in a crash, its a telemetry log

Here is the log:

https://drive.google.com/file/d/1azypx8iu37m7Bs42ivyHFmQcJiVAQBec/view?usp=sharing

Any help is gladly appreciated!!