DOV-Vlaanderen / groundwater-logger-validation

Analysis on validation methods for groundwater logger data
MIT License
2 stars 2 forks source link

levelshift not detected #68

Open fredericpiesschaert opened 3 years ago

fredericpiesschaert commented 3 years ago

there is a clear level shift in this series but it is not detected. Data were uploaded here: https://github.com/DOV-Vlaanderen/groundwater-logger-validation/commit/825d1b53578d99f20f035917b611b783a82c31a2

image

DavorJ commented 3 years ago

Nice example.

image

These are 1h intervals. This is a-priori probability density for 1h intervals:

image

A drop of around -37 is barely in range. Our a-priori data is based on Geotech data which contained tidal effects. (Cf. documentation)

For 1h intervals, a tide effect is possible, hence broad confidence intervals. If I recall correctly, most of the INBO divers we tested the algorithm on were 12h intervals? As you can see below, for 12h intervals there is (almost) no tidal effect and CI is even smaller than for 1h intervals. (If we would have taken a "no-tidal-effect" CI of lets say +/-10 for 1h intervals, then this would have certainly been detected as a levelshift.)

image

As it is now, there is no way to determine if a series has tidal effects or not. (This is a future work option.)

Going further... assuming I discard 11 of the 12 sequential points to simulate a 12h series, even then it is not detected as a levelshift because it is barely within the 12h apriori interval... sigh! (That is: we have already seen such high jumps in the a-priori data.)

image

# sample code for 12h interval simulation
df <- gwloggeR.data::read('KBRS014X_151031')$df
gwloggeR::detect_levelshifts(x = df$PRESSURE_VALUE[(1:nrow(df))%%12==1], timestamps = df$TIMESTAMP_UTC[(1:nrow(df))%%12==1], 
                             apriori = gwloggeR::Apriori('hydrostatic pressure'), plot = TRUE)

So yes, lots to say about this one... I'll set it as a feature request and leave it open for once we decide to revise tidal effects and a-priori information.