search

wandnz / streamevmon

Framework and pipeline for time series anomaly detection
GNU General Public License v3.0
1 stars 1 forks source link

InfluxSourceFunction misses measurements if large history requested #38

Closed wandgitlabbot closed 3 years ago

wandgitlabbot commented 3 years ago

In GitLab, by Daniel Oosterwijk on 2020-11-12

During startup, the InfluxSourceFunction requests all the historical data it was told to. This can take a while, depending on how much history has been requested. It then jumps immediately into creating a subscription, which will only be sent new data that is added to the database after the subscription is added. There is a blank spot which occurs during the historical data gathering where any new data won't be obtained, despite the timestamp of the last gathered measurement being kept track of. This can be quite significant on larger datasets.

wandgitlabbot commented 3 years ago

In GitLab, by Daniel Oosterwijk on 2020-11-16

Fixed in 9d969d87 of traceroute branch.

When the historical data-gathering finishes, we start it again with the start time set to the old end time, meaning any new data that came to the database in the meantime is obtained. This is repeated until there is no new data. This might happen continuously in cases where the RTT between this program and the database is high, and the frequency of new measurements is very high. This just means that the source is polling instead of subscription though, which is fine. All the data should still get through.