search

StackStorm / st2

StackStorm (aka "IFTTT for Ops") is event-driven automation for auto-remediation, incident responses, troubleshooting, deployments, and more for DevOps and SREs. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see https://exchange.stackstorm.org) and ChatOps. Installer at https://docs.stackstorm.com/install/index.html
https://stackstorm.com/
Apache License 2.0
6.08k stars 749 forks source link

Expose metrics API for Prometheus #4341

Open alexandrejuma opened 6 years ago

alexandrejuma commented 6 years ago
SUMMARY

In #4004 the prometheus api was in scope (for validation) but the discussion went cold somewhere along the way.

This feature request is specifically for the prometheus compatible metrics api

ISSUE TYPE
EXPECTED RESULTS
ACTUAL RESULTS
Kami commented 6 years ago

It's on our long-term roadmap, but it will require bigger changes / refactor in our service structure so we can accommodate that (each service will need to expose an HTTP interface).

In the mean time you can perhaps use something like this - https://github.com/prometheus/statsd_exporter.

kedare commented 4 years ago

Any update on this ? I think the first thing that could be interesting would be to have a method that generate a proper metrics name with the proper keys (vs what is done currently what fuse all the keys with the metric name and make it hard to get the information back). Once this part is coded it should be quite easy for other people to create metrics adapters to push to Prometheus/InfluxDB, etc...

nmaludy commented 4 years ago

@kedare i saw that you sub submitted an st2docs PR for configuring Telegraf / InfluxDB. As a quick fix, i wonder if we could use the Telegraf prometheus_client (https://github.com/influxdata/telegraf/tree/master/plugins/outputs/prometheus_client) plugin to expose metrics for Prometheus?

kedare commented 4 years ago

Indeed I did a PR as workaround using Telegraf :)

I am not sure this is easily doable thru Telegraf for Prometheus as Telegraf has much better support for InfluxDB (they are both part of the TICK stack), also Prometheus doesn't have the concept of fields that InfluxDB has (so we could not reuse the same template rules than the ones for InfluxDB)

However Prometheus has the statsd_exporter that does support this type of mapping: https://github.com/prometheus/statsd_exporter#metric-mapping-and-configuration

winem commented 4 years ago

Actually I'm working on a blog post (that comes with a short extension to the st2-docs once it's all working as expected) where I'm using the prom/statsd-exporter container to receive the statsd metrics and expose them to be scraped by a prometheus server. First tests look good and this seems to be a "low hanging fruit" when you want to use Prometheus and have a Docker/Kubernetes setup.

arm4b commented 4 years ago

Thanks to @winem, there is a dedicated documentation section with configuration example about Pushing metrics to Prometheus via the statsd_exporter