Implementation of "station outlier filter"

[cchwala]

Draft code in Python

We have a first very preliminary draft of the code for the "station outlier filter" here at the bottom of the notebook.

Some notes from discussion with Lottte:

• Basic reasoning of the filter: The filter looks at min. two weeks of data (for 5-minute data) at evaluates the correlation with neighboring stations
• Two weeks are chosen because short enough to also apply on a "event basis", because first two weeks cannot be evaluated
• R tricks to speed up things:
  • original R code is here
  • use apply in combination with which in R, e.g. like in line 83 in the linked code

    SOflag[which(apply(cortable, 1, function(x) median(x, na.rm=T)) < corthres)] <- 1 

• doing with a fixed window length is also reasonable, but maybe limits applicability when studying short events

[lepetersson]

[cchwala]

One remark regarding the discussion about potential to improve computation speed: I now found the comment which I referred to in our discussion today when I said that I recall to have read somewhere about an R package to speed things up. The comment is in this notebook of the training school in the "Notes" somewhere at the top of the notebook. This is the comment

-It is possible to run the code faster using the sf package and a maximum limit for the number of neighbour PWSs (if more nearby stations are available only the closest neighbours are considered). See PWSQC_1_alternativeversion.R for that approach

Hence, this only concerns the calculation of the nearby neighbours, for which, if needed, we can also use something like scipy.spatil.KDTree.

[cchwala]

implementation and further discussion is ongoing in #26