search

ArduPilot / ardupilot

ArduPlane, ArduCopter, ArduRover, ArduSub source
http://ardupilot.org/
GNU General Public License v3.0
11.01k stars 17.55k forks source link

5525 airspeed i2c noise #7188

Closed tridge closed 6 years ago

tridge commented 7 years ago

We need a way to filter noise on the MS5525 airspeed sensor. On the 4425 we can use double reads to detect errors. On the MS5525 we can't do that, as a 2nd read will always give zero. This is a log of a flight leading to a stall and crash due to extreme i2c noise. ms5525-airspeed-noise

Log is here: http://uav.tridgell.net/DaleRogers/00000021.BIN

magicrub commented 7 years ago

what about comparing current sample to a high-passed previous sample? Or if delta between samples is large, use previous sample.

kd0aij commented 7 years ago

Are these errors solely attributable to bad I2C wiring/routing, or is there a problem with the voltage levels on the bus?

WickedShell commented 7 years ago

@kd0aij looks similar to the problems I was having with the MS4525 which were all I2C interference related: See here and here (second link has a scope trace of what the bus looks like during one of the bad reads)

kd0aij commented 7 years ago

If this data is from an improperly designed sensor, this might be the root cause:

https://github.com/PX4/Hardware/issues/55

WickedShell commented 7 years ago

@kd0aij it wasn't, was from a MRo and jDrones airspeed + compass board. Both exhibited the same noise for me (as did a 3DR one).

kd0aij commented 7 years ago

Were you unable to eliminate the noise by shortening or shielding the I2C cable?

WickedShell commented 7 years ago

Not possible due to mounting constraints on the first one, the second didn't seem to have any useful effect, however all shielding was an improve affair and not great. Also tested the differential encoder pair from MRo but didn't seem to see any benefit from it. (It was tested before I had identified the actual line level problem). It's worth noting that the log this issue is open from is not mine, and the pair of PR's I linked you completely resolved all my I2C airspeed problems.

kd0aij commented 7 years ago

Your second link is circular, so I haven't seen your scope trace, but this is definitely a good reason to move to CAN for larger aircraft. But I've not observed this problem at all in my little E-flite Convergence

WickedShell commented 7 years ago

@kd0aij oops! Sorry, I've updated the link, it was PR #5950 My setup had a long I2C cable (somewhere on the order of 20-30 cm's) and it runs along a RF cable and is quite close to the actual antenna on a RFD900 (basically it's a terrible spot for wiring reasons, but mounting reasons meant it was the best option available).

lucamax commented 6 years ago

I'm using the Mro Ms5525 sensor since a while but do not ever saw any problem. I'm using a Mro X2.1 Rev. 2 board with 20 cm I2C cable. gkrellshoot_12-10-17_203737

DavidIngraham commented 6 years ago

image image We have been seeing this issue on several of our vehicles as well. We have a custom avionics board with the MS5525 mounted on the same circuit board as the Cube, so I'm surprised that electrical interference is affecting it. Sometime we see it infrequently (the top photo is a two hour flight) but on another it got so bad we had to abort the flight.

I support the idea of sanity checking against past values. Rejecting sensor measurements with a delta of over ~30 m/s over an average of previous readings would fix the issue that we are seeing.

tridge commented 6 years ago

@DavidIngraham wondered if it could be a software issue, so I've created a branch here that logs the raw data from the I2C bus: https://github.com/tridge/ardupilot/commits/plane38-ms5525-logging the log will allow us to check the software side and ensure we're not doing something like mixing up P and T values

DavidIngraham commented 6 years ago

image When investigating the log further we discovered that the "noise" was alternating between the temperature and pressure readings. This seems to suggest that there may be a bug in the driver contributing to the noise we are seeing.

We are going to attempt to reproduce this issue on the ground using the fork @tridge linked above.

tridge commented 6 years ago

I've created a PR to fix this: https://github.com/ArduPilot/ardupilot/pull/7440 many thanks for the log from @DavidIngraham showing that it was a timing issue and not bus noise

custeve commented 5 years ago

@tridge @magicrub - Running 3.9.6, I may have reproduced the same or similar issue. Investigating a possible bad trace on a custom avionics board. Nonetheless, this flight caused a forced landing. See attachment, MS5525 reports noisy readings, then holds. Log file is >10mb, so I couldn't upload.

image

lukedempsey commented 5 years ago

I have recently experience similar intermittent issues but with a 4425.

The artifacts I'm seeing are in the scope trace below. It only seems to happen when the SCL is already low and the SDA line goes high.

Have tried with multiple pixhawks and multiple 4425 sensors.

MVIMG_20190618_132423_2