ioos / ioos_qc

:ballot_box_with_check: :ocean: IOOS QARTOD and other Quality Control tests implemented in Python
https://ioos.github.io/ioos_qc/
Apache License 2.0
46 stars 27 forks source link

Flat-line test rewrite #11

Closed jessicaaustin closed 5 years ago

jessicaaustin commented 5 years ago

This PR is a complete rewrite of the Flat Line Test.

This rewrite includes:


Algorithm Changes

The current implementation compares the current point n to a number of previous observations, and flags it if those observations are within a certain tolerance. This follows what the QARTOD manual says (pg 18):

This test compares the present observation (n) to a number 
(REP_CNT_FAILor REP_CNT_SUSPECT) of previous observations. 
Observation n is flagged if it has the same value as previous observations 
within a tolerance value, EPS. 

However, this can lead to un-intuitive results in some cases.

Take this example:

"flat_line_test": {
    "tolerance": 1,
    "suspect_threshold": 43200,  # 12h
    "fail_threshold": 86400      # 24h
  }

Current implementation: Selection_173

It doesn't seem right that the test can go from FAIL to PASS to FAIL again, without a SUSPECT transition in between.

This is happening because in the current implementation, we create an envelope around the current point, based on the threshold size. If all the points in the chunk are in that envelope, then it fails the test. If any of the points in the chunk are outside the envelope, it passes. Selection_170 Selection_171 Selection_172

There's another way to do this: Use a rolling window, with endpoint at point n, and calculate the range of values in the window (using abs(max-min), or "point-to-point", method). If that range is within tolerance then the point is flagged. Selection_174

Selection_174_edit_fail Selection_174_edit_pass Selection_174_edit_pass_again

In this example, after it goes over the "hump", the range of values in the window 24hrs before point n still exceeds the threshold, so the point passes the test.

For more "traditional" flat line scenarios, the two implementations behave exactly the same: Selection_168 Selection_175


Note: this PR supersedes the following PRs:

jessicaaustin commented 5 years ago

@kwilcox @eldobbins please review!