This is a set of changes that allow the monitoring stack to be configured to send metrics to a remote Prometheus receiver, both with our current Prometheus monitoring type and with a new PrometheusAgent type that deploys a more minimal Prometheus installation only for scraping and sending metrics.
Prometheus Agent mode
The Agent mode optimizes Prometheus for the remote write use case. It disables querying, alerting, and local storage, and replaces it with a customized TSDB WAL. Everything else stays the same: scraping logic, service discovery and related configuration. It can be used as a drop-in replacement for Prometheus if you want to just forward your data to a remote Prometheus server or any other Remote-Write-compliant project.
Notice that with Agent mode:
There's no much to do form Prometheus UI, no need for exposing it via ingress.
There's no need for Grafana (because it can't query Prometheus for the data).
Alertmanager that not work with agent mode (it can't query Prometheus).
❗ kubectl top and similar won't show resources usage, because prometheus-adapter does not work with agent mode (it can't query prometheus for the metrics).
If any of the previous points is a blocker for a particular use-case, say you want to be able to see resources usage with kubectl top, you can fallback to the regular prometheus monitoring type and configure the remoteWrite section setting a low retention to minimise resource usage instead of using the prometheusAgent type.
Changes in this PR
Add remoteWrite section to prometheus' configuration in the schema, so Prometheus can be set up to send metrics to a remote receiver.
Add prometheusAgent as monitoring type, as an alternative to prometheus+remoteWrite, allowing to have a "minimal" monitoring stack capable of sending metrics to a remote Prometheus receiver. Notice that this mode does not allow for querying the local Prometheus instance (so no local Grafana dashboards) or having local alerting set up.
Add migrations for switching between Prometheus and Prometheus Agent, and to and from none.
Intro
This is a set of changes that allow the monitoring stack to be configured to send metrics to a remote Prometheus receiver, both with our current Prometheus monitoring type and with a new PrometheusAgent type that deploys a more minimal Prometheus installation only for scraping and sending metrics.
Prometheus Agent mode
The Agent mode optimizes Prometheus for the remote write use case. It disables querying, alerting, and local storage, and replaces it with a customized TSDB WAL. Everything else stays the same: scraping logic, service discovery and related configuration. It can be used as a drop-in replacement for Prometheus if you want to just forward your data to a remote Prometheus server or any other Remote-Write-compliant project.
Notice that with Agent mode:
kubectl topand similar won't show resources usage, becauseprometheus-adapterdoes not work with agent mode (it can't query prometheus for the metrics).If any of the previous points is a blocker for a particular use-case, say you want to be able to see resources usage with
kubectl top, you can fallback to the regularprometheusmonitoring type and configure theremoteWritesection setting a low retention to minimise resource usage instead of using theprometheusAgenttype.Changes in this PR
remoteWritesection to prometheus' configuration in the schema, so Prometheus can be set up to send metrics to a remote receiver.none.Refs
fixes https://github.com/sighupio/product-management/issues/484