reckon

Reckon is a prometheus exporter for time series forecasting and anomaly detection. It scrapes prometheus metrics with a defined time range, train a predictive model with those metrics, and then expose the original and forecasted metrics back through a prometheus HTTP endpoint.

An exmaple of original and forecasted metrics generated from Prophet and Tangram gradient boosting model and then exposed from reckon:

# HELP sensehat_humidity_prophet Prophet forecasted metric value
# TYPE sensehat_humidity_prophet gauge
sensehat_humidity_prophet{column="yhat",instance="sensehat.rpi.topbass.studio:8000",job="sensehat_exporter"} 58.246349893997056
sensehat_humidity_prophet{column="yhat_lower",instance="sensehat.rpi.topbass.studio:8000",job="sensehat_exporter"} 57.4450973695544
sensehat_humidity_prophet{column="yhat_upper",instance="sensehat.rpi.topbass.studio:8000",job="sensehat_exporter"} 59.08650789641951
# HELP sensehat_temperature_prophet Prophet forecasted metric value
# TYPE sensehat_temperature_prophet gauge
sensehat_temperature_prophet{column="yhat",instance="sensehat.rpi.topbass.studio:8000",job="sensehat_exporter"} 22.196715588009724
sensehat_temperature_prophet{column="yhat_lower",instance="sensehat.rpi.topbass.studio:8000",job="sensehat_exporter"} 21.83335916260913
sensehat_temperature_prophet{column="yhat_upper",instance="sensehat.rpi.topbass.studio:8000",job="sensehat_exporter"} 22.551863453282152
# HELP sensehat_humidity_tangram Tangram forecasted metric value
# TYPE sensehat_humidity_tangram gauge
sensehat_humidity_tangram{column="yhat",instance="sensehat.rpi.topbass.studio:8000",job="sensehat_exporter"} 58.43156051635742
sensehat_humidity_tangram{column="yhat_lower",instance="sensehat.rpi.topbass.studio:8000",job="sensehat_exporter"} 54.96922722581767
sensehat_humidity_tangram{column="yhat_upper",instance="sensehat.rpi.topbass.studio:8000",job="sensehat_exporter"} 61.893893806897175
# HELP sensehat_temperature_tangram Tangram forecasted metric value
# TYPE sensehat_temperature_tangram gauge
sensehat_temperature_tangram{column="yhat",instance="sensehat.rpi.topbass.studio:8000",job="sensehat_exporter"} 21.94704246520996
sensehat_temperature_tangram{column="yhat_lower",instance="sensehat.rpi.topbass.studio:8000",job="sensehat_exporter"} 20.179807771933568
sensehat_temperature_tangram{column="yhat_upper",instance="sensehat.rpi.topbass.studio:8000",job="sensehat_exporter"} 23.714277158486354

Each metric configured in reckon will have its forecasted metric generated and exported in 3 dimensions (labels):

• yhat: forecasted value
• yhat_upper and yhat_lower: uncertainty interval

Try it

Gather the following info before start:

• Prometheus server address, for example, http://prometheus.rpi.topbass.studio:9090
• Metric names to watch and models for each metric, accepts inline YAML or comma separated list, for example, {sensehat_temperature: [Prophet, Tangram], sensehat_humidity: [Prophet, Tangram]} or sensehat_temperature,sensehat_humidity

With Docker

This is the simplest method to get reckon running. You only need docker to get started.

PROM_CLIENT_URL={prometheus_server_address} \
WATCH_LIST={inline_yaml_or_comma_separated_list} \
make docker

With Go, Python and Virtualenv

If you prefer running reckon without docker, or you would like to build and run the binary locally, you will need Go, Python and Virtualenv. Make sure you have Go 1.16+ and Python 3.7

• Go: brew install go or follow this gudie
• Pyenv: brew install pyenv or follow this guide
• Python3.7
  • find a Python 3.7 version with pyenv install -l | grep '^\s*3.7'
  • select a version (eg. 3.8.12) and install it with pyenv install 3.8.12
  • switch to the installed version pyenv global 3.8.12
• Virtualenvwrapper: pip install virtualenvwrapper AND follow this guide

PROM_CLIENT_URL={prometheus_server_address} \
WATCH_LIST={inline_yaml_or_comma_separated_list} \
make run

Configure it

Reckon can be configured with the following environment variables:

WATCH_LIST accepts either inline YAML or comma separated list, for example:

• {sensehat_temperature: [Prophet, Tangram], sensehat_humidity: [Prophet, Tangram]}
• sensehat_temperature,sensehat_humidity
  • When using comma separated list, MODELS becomes a required field

Available models:

• Prophet
• Tangram with gradient boosting tree model
• Future plans: add ARIMA and LSTM models

Use it

• Find the latest release and downlaod a .tar.gz archive for your OS and architecture.
• Generate root CA, server and client keys and certificates for gRPC mTLS calls to Python ML models. Here is an example for generating the keys and certs with certstrap.
• Deploy reckon to bare metal hosts. Here is an example with ansible.
• Alternatively, build a debian slim based docker container, and deploy the container to kubernetes or other orchestration platforms.

Monitor it

In addition to forecasted metrics, reckon also exposes runtime metrics to help monitor reckon itself.

Reckon exposes build info through reckon_build_info

# HELP reckon_build_info Information about reckon build.
# TYPE reckon_build_info gauge
reckon_build_info{commit="c5159d0375d59b0255069eeec3fe3d3ccfe42207",date="2021-11-24T08:05:34Z",go_version="1.16.10",goreleaser_version="0.174.0",python_version="3.8.12",version="0.5.2"} 1