Anomalous data detection

[catbarph]

Basic outlier detection is needed to identify & filter out really bad data. For instance: temperature sensors will never actually read over a few hundred degrees (but they have been spiking to 6K repeatedly!), so it should be easy to detect/throw out values outside of that range. Note: i2c collisions may have been the culprit...

[JakeHartnell]

Agreed, we should have some utilities for detecting when sensors are malfunctioning. When you have a temperature that's reporting 6,000 C something is obviously wrong.

[catbarph]

Where did you get to re: low-pass filtering?

[JakeHartnell]

There are actually a decent amount of open source libraries for this:

• https://github.com/nathanielc/morgoth
• https://github.com/twitter/AnomalyDetection
• https://github.com/uhho/anomaly-detector
• https://github.com/KenanSulayman/anomaly
• https://github.com/sergibondarenko/anomaly-finder

[catbarph]

Still seems to mainly be an issue for the Pt100 water temperature sensor, connected over SPI.

[JakeHartnell]

I was thinking about this on the plane, this issue is mislabel as we can easily detect anomalies now. What's wanted is the ability to filter out anomalies.

This library and Thing.js are in need of some re-architecture if we are to resolve issues like this. As it stands, we can emit alerts just fine, but if we are to filter data we have a problem in that Grow.js itself listens for these events to be emitted (and after they're emitted, it's already to late for filtering as those values will hit the server and F@#$ up timeseries charts).

When Thing.js (an extension of the event emitter class) emits an event, we extend the emit function so that an optional hook function is called and ether returns a truthy value, or returns nothing, in which case we emit the event. If a hook function returns false, we DO NOT emit the event. I'm not sure how I feel about an approach like this.

There might be better ways of achieving similar functionality??? Perhaps piping events in an interesting way that allows them to potentially be intercepted?

Additionally, rather than the above approaches, we could make a helper function that checks whether a value is in bounds. This would be used by implementers when creating a driver as the following pseudo code illustrates:

let grow = require('Grow.js');
grow.registerTargets({
  temperature:{
    min: 20,
    max: 80,
    bounds: [0,100]
  }
})

let value = Math.random() * 200;

if (grow.inBounds('temperature', value)) {
  grow.emit('temperature', value);
}

This introduces extra complexity though, but in a way that isn't "magic" (ie. the developer explicitly chooses to filter values).

[JakeHartnell]

To address this issue there is a new inBounds() method in Grow.js v0.4.7, to be used as above. We may create a list of default sensible bounds for common sensor types so that they don't have to be defined in a Growfile.