This is likely only worth to implement when we have individual qc test results saved.
Scope
At the moment, we don't do any official checks for the time sampling values. For example, there is no TIME_quality_control variable. Hence, any sample step that is shifting from the default "instrument_sampling_interval" (or burst for that matter), is not evaluated.
Problems
Given that most of the SpikeQC tests take into consideration differences (between points or within a range), this may be a problem for detection and generate false-positives. A typical scenario is on the application of tests in a burst sampling data, where the number of samples in the burst are departing from the specification of time. If Incomplete bursts are present, an increase/decrease in time distance between them will affect statistics. A typical example is the time shifts towards the end of time-series in some WQMs. Other rare causes are exotic time shifts in the middle of the series. I noticed some significant amount of time shifts in some burst instruments.
Proposed Solution
TimePrecisionQC:
[ ] Mark the start of the first time-shift for reference.
[ ] Mark all subsequent time-shifts.
[ ] Use a fix threshold for detection (e.g. 1second).
[ ] Allow user to mark everything after the time-shift to be bad (non-burst), the respective burst, or
[ ] Show a window with the time-shift markings and allow the user to manual select/brush the bad time values.
[ ] Maybe look into battery values to estimate possible instrument time shifts.
[ ] Create flag comment templates ("time precision loss", "large time shift" , "unequal burst").
SpikeQC:
[ ] Allow dependency on TimePrecisionQC as required.
Further Requirements:
[ ] Collect test cases with different time sampling problems (jumps/precision loss/ different burst lengths/etc). [wip]
[ ] TIME_quality_control definitions for every possible problem in the TIME values (large time shifts, unequal bursts, loss of precision)
[ ] How to handle the TIME_quality_control variable in some despiking tests? Split the series, enable despiking tests to use it?
Updated:
This is likely only worth to implement when we have individual qc test results saved.
Scope
At the moment, we don't do any official checks for the time sampling values. For example, there is no
TIME_quality_controlvariable. Hence, any sample step that is shifting from the default "instrument_sampling_interval" (or burst for that matter), is not evaluated.Problems
Given that most of the SpikeQC tests take into consideration differences (between points or within a range), this may be a problem for detection and generate false-positives. A typical scenario is on the application of tests in a burst sampling data, where the number of samples in the burst are departing from the specification of time. If Incomplete bursts are present, an increase/decrease in time distance between them will affect statistics. A typical example is the time shifts towards the end of time-series in some WQMs. Other rare causes are exotic time shifts in the middle of the series. I noticed some significant amount of time shifts in some burst instruments.
Proposed Solution
TimePrecisionQC:
SpikeQC:
Further Requirements: