search

influxdata / kapacitor

Open source framework for processing, monitoring, and alerting on time series data
MIT License
2.31k stars 492 forks source link

Make snapshots work even if the task DAG changes #775

Open nathanielc opened 8 years ago

nathanielc commented 8 years ago

Currently snapshots are associated with tasks and node snapshot data is keyed by the node ID from the DAG. These node IDs can easily change. We need a way to make saving/loading a snapshot work even if the DAG changes.

nathanielc commented 7 years ago

The solution to this is proposed to make use of a new mechanism called the blob store. See https://github.com/influxdata/kapacitor/blob/master/BLOB_STORE_DESIGN.md

khoxsey commented 7 years ago

I am interested in the use case mentioned in the design document, to load and store trained model data. In particular, there are families of filters built from time-series models (the many variations of AR and MA, for example) in which the runtime execution is a relatively simple computation of weights and a moving window of values.

There is a class of problems of this type that lend themselves well to representation in InfluxDB. The input data arrives from the outside and is written to the database. The points themselves can be streamed through a filter UDF running in Kapacitor, providing realtime analytics. And then the base input data can be collected in batch and re-estimated, allowing a runtime system to easily perform its own rolling updates.

The features to implement this missing from the TICK stack right now are:

The Blob Store proposal appears intended to provide opaque storage/retrieval for a UDF. In terms of providing a robust snapshot store to aid in system startup, and thus avoid a crippling amount of startup-querying as a bunch of big filters queue up their history data, it seems like a good approach.

However, as a mechanism for supporting the filtering models mentioned above, it seems like a step away from the easy transparency of the time series data stored in InfluxDB. While it is easy to imagine an external program performing a model estimation and storing the new weights into a blob for the UDF to read, if one wants to store the weights as-computed and as-used it seems more in-character to store the data as Points.