Prometheus endpoint for internal stats

[sklirg]

We're using Prometheus for our metrics. I know that Skogul supports internal stats, and can send it through any sender to a valid recipient. However, Prometheus is pull-based, rather than send-based (which skogul is), so I can't really see any simple way of exposing these metrics.

This is an open question of supporting more ways of exposing skogul internal metrics (in this case: prometheus).

An example implementation if this is accepted could be to ship a startup-flag for a dedicated http server which exposes these metrics, pulled from the internal skogul.Stats structure(s), e.g. using -prometheus-metrics-listen-addr, -prometheus-metrics-listen-port, etc... This way it will not interfere with skogul unless explicitly enabled.

[KristianLyng]

I have been meaning to write a http sender that's poll-based, which keeps the latest metrics of some sort and exposes them on demand, to support hooking in to Prometheus.

This would be a more general solution, and I don't think it's that difficult to write. Give me a few weeks and I'll see if I can't throw something together.

[HON95]

pushgateway is an alternative way of handling this scenario, to avoid feature-creeping pull-based collection semantics into a push-based solution.

(Stumbled upon this while procrastinating.)

EDIT: Remote write is another solution which might be more appropriate if implemented properly.

[holene]

Meteorologisk institutt calling, just checking the status on the Prometheus endpoint. That would solve a lot of problems for us 😊

[HON95]

Just commenting to withdraw my suggestion from 1.5 years ago.

While using push-based toward Prometheus is an option that could be cleaner for the Skogul side, it's a suboptimal hack seen from the Prometheus side (as probably mentioned in the both those links). A configurable metrics endpoint as an implicit feature flag and using the Go client lib would be much more "standard". Or just generating the Prometheus timeseries manually using the plaintext exposition format to avoid the dependency, but then you wouldn't get e.g. automatic Go process metrics or the more efficient protobuf exposition format for free.