QubitProducts / exporter_exporter

A reverse proxy designed for Prometheus exporters
Apache License 2.0
336 stars 55 forks source link

manipulate pushgateway through this exporter #29

Closed bennydubois closed 4 years ago

bennydubois commented 4 years ago

I have many nodes that should push and delete metrics to pushgateway on a remote server. Since the labels are parsed from the url, I am wondering if it is possible to do it through exporter exporter or if I should access pgw directly.

Thanks

tcolgate commented 4 years ago

I'm not really sure what exporter_exporter could add here. Do you want to proxy to the remote pgw and add labels to the pushed metrics? It's not the kind of use we really evisioned, but if you can make a really compelling use case, and can think of a pleasant configuration format, I'd consider it.

bennydubois commented 4 years ago

We are using cloud instances with autoscaling groups on different providers, that can increase in size from 20 to about 400 instances on a daily basis. We can't use the provider's service discovery since part of the requirements are that it should stay platform agnostic so we are using consul.

However, sometimes instances fail start properly: won't join either consul or our service infrastructure and we end up with an unmonitored idling server. Until that's fixed I coded a small service to monitor consul agent's status and report failures to a pushgateway located on a metal prometheus server with a static IP. I would rather go through exporter exporter to add and remove this data to keep our operational standard and avoid opening more ports and creating a precedent that it's okay to not go through the proxy.

If it isn't possible I'll open the port and access pgw directly, but I wanted to make sure first.

tcolgate commented 4 years ago

It sounds like this would be using expexp as a straight up reverse proxy. I realise it's not always obvious that it shouldn't be, but being a proper reverseproxy would be a much more complicated task. If you can find a syntax for configuration, and some way that expexp can add value, I'll happily review a PR.