Open cregouby opened 3 years ago
*_gaps()
don't know how to handle irregular temporal data. {tsibble} doesn't support dual index. index2
means temporary grouping. Can you please elaborate on what outcome is expected for an irregular tsibble?
Hello @earowang, Sure *_gaps() don't know how to handle irregular temporal data, and this is the reason for my feature request here. And sorry for my mistake in using index2 as a secondary index for the example.
Irregular time series with start-stop / duration are very often used in process control, industrial robot monitoring, ... So detecting gaps can be crucial on those datasets. As tsibble is a fantastic framework, I would love to have it extended to irregular time-series with start-stop, as it is a generalisation of the tsibble interval in the calculation of gaps.
Expected outcome is, like in the provided example with nyc_bikes
, all start-stop interval where data is missing. This allow usage ratio, efficiency valuation, communication loss detection, ... nyc_bikes
here is a toy example, as usual stat-stop data have contiguous time intervals.
The concept of tsibble's interval is different from {lubridate} start-end Interval
. tsibble's interval is more of time differences between time indices, but lubridate's Interval
defines specific start-end timestamps. A regular time series means a constant difference is assumed across all time indices, and *_gaps()
therefore.
Using nyc_bikes
as an example, the time index can be represented in lubridate's Interval
class. I'm not sure what's the time difference here for x
. Should it be 1 second by aligning start
or end
for these two observations? Or should it be 1 second from the difference between start
and end
? In the field you work on, what's the common practice?
library(lubridate)
obs1 <- interval(ymd_hms("2018-02-26 19:15:40"), ymd_hms("2018-02-27 07:52:49"))
obs2 <- interval(ymd_hms("2018-02-27 07:58:13"), ymd_hms("2018-02-27 12:03:27"))
x <- c(obs1, obs2)
x
#> [1] 2018-02-26 19:15:40 UTC--2018-02-27 07:52:49 UTC
#> [2] 2018-02-27 07:58:13 UTC--2018-02-27 12:03:27 UTC
Created on 2021-09-29 by the reprex package (v2.0.1)
I fully agree of the missleading use of term interval here, so let's call it the min_gap
, that could be the minimum time between end-time of event n and start-time of event n+1 to consider it been a gap
.
In the field I work on, the alignment between start-time and next end-time depends on each situation, and is usually linked to the source timestamp precision. ( This is not a very usefull statement, I know) We can only assume that the gap cannot be raised for less than min_gap<1
(of the last digit of the time precision, i.e. no less than one seconds for ymd_hms()
timestamps)
The two main use-cases are
min_gap
of 1 second. But we could think of a parameter to force that min_gap
to be higher, when the times are known to be the result of rounding operation like "closest 10 second step" on slow agents/machines.min_gap
is strictly 1, but may be relaxed to a some higher value to allow some clock offset between agents,...That makes me think of a configurable min_gap
parameter in the *_gaps
functions for irregular start-end timeseries...
Brief description of the problem
*_gaps
are a very usefull set of commands. And I would love to be able to use them on irregular longitudinal data withstart
andstop
index columns, liketsibbledata::nyc_bikes
.Currently, irregular interval tsibble are not supported by
*_gaps
function familyWhat output is expected
This is a manual edit :