IOOS_QC QARTOD Spike_Test is a direct implementation of the spike test described within the QARTOD Manual. This algorithm is acting in some conditions similarly than the the rate_of_change test. We present here a potential implementation of an alternative method which will target exclusively the single spike values over a certain threshold.
I suggest implementing this second method as an option within the QARTOD spike_test via a method optional input and keeping the original average method as the default one.
The method consider essentially that a value N is flagged if the following conditions are respected:
The absolute value of the difference between the value N and its preceding (N-1) and following value (N+1) is bigger than a threshold
The multiplication of both difference is smaller than 0 (opposite direction).
""
To highlight the difference between the two methods, an example can be find here within a jupyter notebook which is using the following vector and threshold values:
By applying the regular QARTOD Spike method, we get the following result:
Differential Method (suggested alternative method)
Here's the results in similar conditions for the suggested differential method.
As shown, the second method only flag the single spike values without flagging records that are associated with a strong preceding jump. The QARTOD rate of change test can then be use to flag those data more rigorously if desired.
kwilcox
commented
3 years ago
IOOS_QC QARTOD Spike_Test is a direct implementation of the spike test described within the QARTOD Manual. This algorithm is acting in some conditions similarly than the the rate_of_change test. We present here a potential implementation of an alternative method which will target exclusively the single spike values over a certain threshold.
I suggest implementing this second method as an option within the QARTOD spike_test via a method optional input and keeping the original average method as the default one.
The method consider essentially that a value N is flagged if the following conditions are respected:
"(X_{N}-X_{N+1})<0\;&space;\text{Flag}\Rightarrow&space;X_N "\text{if}\; \begin{matrix} |X_{N-1}-X_{N}|\\ |X_{N}-X_{N+1}| \end{matrix} > \text{threshold} \;\land\; \(X_{N-1}-X_{N}\)(X_{N}-X_{N+1})<0\; \text{Flag}\Rightarrow X_N")
"
To highlight the difference between the two methods, an example can be find here within a jupyter notebook which is using the following vector and threshold values:
Average Method (suggested by QARTOD)
By applying the regular QARTOD Spike method, we get the following result:
Differential Method (suggested alternative method)
Here's the results in similar conditions for the suggested differential method.
As shown, the second method only flag the single spike values without flagging records that are associated with a strong preceding jump. The QARTOD rate of change test can then be use to flag those data more rigorously if desired.