rate of change test - Githubissues

ioos / ioos_qc

:ballot_box_with_check: :ocean: IOOS QARTOD and other Quality Control tests implemented in Python

https://ioos.github.io/ioos_qc/

Apache License 2.0

46 stars 27 forks source link

rate of change test #44

Open petejan opened 3 years ago

petejan commented 3 years ago

The rate of change test is implemented as delta value / delta time, where as the Manual for Real-Time Quality Control of In-situ Temperature and Salinity Data has

which is a excessive delta greater than the standard deviation of surrounding 25 hrs (as in example 1) of data, which is quite a different metric.

kwilcox commented 3 years ago

Thanks @petejan. Looks like it was changed in https://github.com/ioos/ioos_qc/pull/2 to accommodate some real-world cases we were encountering. I'd be happy to accept a PR that accepts both a computed threshold parameter and the N_DEV / TIM_DEV parameters and acts accordingly depending on which were provided.

JessyBarrette commented 3 years ago

Ideally, the Rate of Change test would need to be applicable along the different axis (time, depth, lat, lon). As of now, it is only compatible along the time axis.

See [QARTOD Temperature and Salinity Manual] p.25

(https://cdn.ioos.noaa.gov/media/2020/03/QARTOD_TS_Manual_Update2_200324_final.pdf)

I would propose adding the optional inputs "axis", "zinp", "lat", and "lon" to the test. The "axis" would define along which axis the test should be applied with a default value along the time axis.

petejan commented 3 years ago

I have been post processing data looking at what the 'best' methods would be to flag data,

as a metric, the rate of change (delta value/delta time) is much the same as the spike test, scaled by the delta time
The test as described in the manual has issues as the threshold changes based on previous data, this has the effect that if the data starts noisy (high standard deviation), the test never flags any data. It works when the data starts ok and gets noisy for some reason.

Not sure what the 'best' method is.

JessyBarrette commented 3 years ago

I found out the same thing regarding the spike test which is in some circumstances acting like the rate of change test. I would personally expect the spike test to detect spikes exclusively. I'm just about to propose something related to this.

@petejan perhaps the best way for you to handle this is use a combination of a running window standard deviation threshold for the small but bad rates of change (when it's not noisy) and a fix threshold for the noisy (high standard of deviation).

If the both type of threshold can be implemented at the same time, I think this would a good solution to your problem.