DOV-Vlaanderen / groundwater-logger-validation

Analysis on validation methods for groundwater logger data
MIT License
2 stars 2 forks source link

flatline detection? #45

Open fredericpiesschaert opened 5 years ago

fredericpiesschaert commented 5 years ago

Two issues:

image

DavorJ commented 5 years ago
  • the flatline at the end is not detected

New function _detectflatline()? One possible implementation would be to calculate the chance that the error term in a random walk process is zero consecutively. (Similar for air pressure as a normal noise process). If highly unlikely (i.e. 1/1E6) that multiple points have the same value, then flag as flatline. Might be tricky to calculate due to sensor rounding of measurements, etc. and that P(X=x) == 0 for normal noise.

  • Does the outlier function take into account that hydrostatic pressure cannot drop below the air pressure minimum? I thought it did, but then everything below let's say 900 should be marked as an outlier.

Indeed, good point. Since change of model to random walk for hydrostatic pressure (e.g. +v0.04), check with apriori air pressure was removed. From a practical point of view, this is not difficult to add back (i.e. all the points below about 900 would be blue then). From a theoretical point of view, there is a big difference between an outlier in a random walk process, and an outlier in a normal noise process. Definitely nitpicking from a certain point of view, but anyhow, that was the reason I removed it :).