influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.23k stars 3.51k forks source link

Build in anomaly detection function in influxdb #6208

Closed dengliu closed 8 years ago

dengliu commented 8 years ago

Feature Request

Proposal: Build in anomaly detection in influxdb

Use case: It would be nice to see holt-winter and some build-in time series anomaly detection functions in influx QL. E.g. https://blog.twitter.com/2015/introducing-practical-and-robust-anomaly-detection-in-a-time-series http://robjhyndman.com/hyndsight/yahoo-data/

With this support, people should be able to see visualize anomalies that happened in the history and be able to set the alert rule more accurately.

Kapacitor for sure is able to integrate with UDF to detect anomaly. However it just reports the results with predefined rule and doesn't give user a good experience of visualizing the anomalies levels over a continuous time span. having buildin anomaly detection in InfluxDB allows people to visualize the anomaly status via viz tools (e.g, Grafana), and thus have a better idea of tunning the anomaly detection system. An example of visualizing anomalies like visualizing regular metric is as below figure_localglobal_anomalies

rossmcdonald commented 8 years ago

I believe the next release of Kapacitor is going to have a built-in anomaly detection function, so I wouldn't expect this to be added to InfluxDB. Kapacitor also has support for User-Defined Functions (UDF's), so custom anomaly detection functionality can be written if needed.

dengliu commented 8 years ago

Thanks for the info @rossmcdonald Kapacitor for sure is able to integrate with UDF to detect anomaly. However it just reports the results with predefined rule and doesn't give user a good experience of visualizing the anomalies levels over a continuous time span. having buildin anomaly detection in InfluxDB allows people to visualize the anomaly status via viz tools (e.g, Grafana), and thus have a better idea of tunning the anomaly detection system. An example of visualizing anomalies like visualizing regular metric is as below figure_localglobal_anomalies

e-dard commented 8 years ago

@dengliu InfluxDB shouldn't be responsible for things like this. The pipeline should be that other tools (e.g., Kapacitor) take InfluxDB data and perform the anomaly detection analysis, and then optionally feed that data back into InfluxDB, which can then be visualised using whatever visualisation tool is in play.