search

joshday / OnlineStats.jl

⚡ Single-pass algorithms for statistics
https://joshday.github.io/OnlineStats.jl/latest/
MIT License
831 stars 62 forks source link

MovingTimeWindow #210

Open JoaoAparicio opened 4 years ago

JoaoAparicio commented 4 years ago

This[1] could be more efficient if the timestamps are assumed to be ordered. TimeSeries.jl enforces this. Do you think it would make sense to have some interop?

[1] https://github.com/joshday/OnlineStats.jl/blob/e554858134f38b3e22499b2de1e4be2e5eab3c67/src/stats/stats.jl#L394-L395

joshday commented 4 years ago

That seems like a fair assumption. The _merge! method would need to change to handle the timestamp ordering.

What kind of interop are you thinking of?

JoaoAparicio commented 4 years ago

Currently we have function _fit!(o::MovingTimeWindow, y). In addition we could implement function _fit!(o::MovingTimeWindow, y::TimeArray). What is currently done is push all values of y into o, then filter. When the new _fit! method is called, timestamps in MovingTimeWindow are guaranteed to be sorted because timestamps in TimeArray are guaranteed to be sorted. So first compute the cutoff point, and then to determine what is kept and what is dropped, use searchsorted (if this point is in y, this is what from(y, cutoff) does, see [1]).

I'm sure this code can be made faster when we know the input array is time-sorted, I just don't know if the cleanest way to implement this is by adding function _fit!(o::MovingTimeWindow, y::TimeArray). Because then that would add TImeSeries.jl as a dependency to OnlineStats.jl, which is annoying.

[1] https://juliastats.org/TimeSeries.jl/latest/split/#from-1

joshday commented 4 years ago

Sorry for the slow response! I'd be happy with changing MovingTimeWindow to assume all of the inputs are sorted. I agree with wanting to avoid a new dependency.

Is this something you'd like to implement? I'm about to have a baby (wife is due today!), so I'll be out of commission for a bit.