sinkingpoint / prometheus-gravel-gateway

A Prometheus Aggregation Gateway for FAAS applications
GNU Lesser General Public License v3.0
115 stars 10 forks source link

Cluster Returning Values from All Peers #23

Open roberth05 opened 1 year ago

roberth05 commented 1 year ago

We have a Gravel Gateway setup in a 3 peer cluster and are pushing metrics via the Python Prometheus Client library. Specifically, we are using push_to_gateway method. This method requires a "job" parameter be set. The push is successful but Gravel is not writing values to the same instance on successive pushes. Over multiple pushes, we end up with 3 different values on the instances.

When scraping, the value returned seems randomly chosen out of the 3 instances.

Expectation: Based on the documentation, the metric should write to the same instance each time and be retrieved from that same instance, regardless of the instance that received the request.