Add support for Pushgateway

[TylerPachal]

Hello,

Is there any plan to add support for sending metrics to a Pushgateway? I have a few use cases for this including:

• batch jobs which may last less than the scrape-interval.
• emitting an application_started counter metric (it is hard to do this right now because if my app crashes on startup before the first scrape the metric is never consumed by Prometheus).

I am not 100% sure where this functionality belongs, I originally started in the prom_ex library and made my way down to here.

I can implement this if someone can point me in the right direction. I believe we just need to get an extra config value into the Counter/Sum/LastValue/Distribution modules' handle_event functions, which we would indicate if the metric should be written to ets like is it now, or sent to the pushgateway. I just don't know where that extra config value should be declared.

[bryannaegele]

Hey @TylerPachal. As far as I know folks run in one configuration or the other, so it would be an instance configuration option and not a per-metric option.

I'm not sure exactly what would be involved in implementing push so if you want to take a crack at it, by all means do. I don't think it removes batching or any other part of this implementation. Push gateways just run batches on a shorter interval to another instance that gets scraped at longer intervals. Docs. It doesn't act like statsd pushing a single synchronous metric on every measurement when using push.

Given all that, any implementation changes should probably be in the other lib since the underlying measurement/aggregation implementation doesn't change.

This library will be replaced by opentelemetry metrics this year. If you're using PromEx, I assume there would have to be work done in that, as well, since it leverages these libs. @akoutmos would be able to weigh in on that.

[akoutmos]

Hey @TylerPachal! It has been a couple years since I last needed to run a Push Gateway in my infrastructure, but iirc, you should be able to just HTTP POST your data to a Prometheus Push Gateway instance (see docs). To get the metrics for the POST, all you should need to do is call the following PromEx function with your PromEx module as the argument: PromEx.get_metrics(YourApp.PromEx). That will return a binary with all of the accumulated metrics up to that point that you can then send to a Prometheus Push Gateway.

Hopefully that helps!

[TylerPachal]

Thanks for the info! For my purposes what @akoutmos described sounds simple enough. Maybe I will open a PR to add a little example to the documentation so that other people can find it as well.