Add flipped-bit tolerance to digital comms

[dcstraney]

Should plan for I2C bus having an occasional bit error from EMI of some kind (nearby ESD, valve spikes, etc.). This means any single sensor reading could have an error:

Proposed handling in software:

• Two readings need to be within X% of each other / within Y offset of each other, to be considered valid (sensor reading frequency needs to be much higher than physical state changes)
• Having X subsequent readings without a valid reading, or having more than Y% of readings be invalid, shows that something is wrong, and either...
  • Triggers the failsafe state
  • ...or the Pi starts using the CPLD's sensor readings, if its own comms have issues and the CPLD's don't? (While displaying a warning/alarm of course: continued operation if possible is probably better than failsafe though?)
• Bonus for later: if using the digital pressure sensors, this error handling could be separated by I2C destination (ADC, pressure sensor 1, pressure sensor 2) to catch failures in a single device vs. general communications - single-device failures could then result in using the redundant reading instead as discussed above

[ralphBellofattoSTX]

@dcstraney how are we planning to deal with the sensor reading with the cpld and the rpi.

• Do all the sensor readings with the cpld *.some kind of multi master scheme for the i2c bus?

if the clpd does it, it can pretty much poll the i2c bus at its highest frequency and do what bascially is a low pass filter with it.

[dcstraney]

Plan is for 2 separate buses actually: had discussed this before with Alan and Stephanie. Having a multi-master bus is possible but needs a bunch of extra logic and hardware to be reliable (preventing a Pi failure or other single device from making the whole bus unusable), and having dual ADCs and pressure sensors (if digital ones used) adds some nice redundancy.

Low-pass filtering the readings makes sense, but depending on filter size, a flipped MSB might still cause a noticeable output shift. Would also be good to be able to explicitly detect errors (either with comms or sensor) to be able to know that something is wrong and possibly switch to redundant bus/sensors.

[ralphBellofattoSTX]

@dcstraney separate i2c busses, so the devices we are using are dual ported?

[dcstraney]

Nope just 2 of each, for redundancy. 2 ADCs, 2 low-pressure sensors, 2 high-pressure sensors...

[ralphBellofattoSTX]

what about the o2 sensor?

[dcstraney]

That's read by the ADCs.