search

jacksontj / promxy

An aggregating proxy to enable HA prometheus
MIT License
1.12k stars 126 forks source link

Multi-tier promxy #226

Closed glightfoot closed 4 years ago

glightfoot commented 4 years ago

Question: is it possible/advisable to setup a multi-tier Promxy deployment? The use case is this: We have multiple k8s clusters in multiple datacenters, each running several HA pairs of prometheus instances. Currently, we have one Promxy instance balancing against all of the load-balanced endpoints of the HA pairs, but this does not let us use promxy to enable HA and gap-less querying. The idea was to have one Promxy in each cluster, using k8s service discovery to discover all the pods, then another Promxy aggregating against all of the cluster Promxys.

Thanks!

jacksontj commented 4 years ago

Great question -- and the answer is Yes! Promxy simply aggregates other prometheus API endpoints together so you can definitely layer promxy (and I know of some that do that all the time). Similarly you can mix prometheus API endpoints -- for example you could have prometheus, promxy, and victoriametrics all as downstreams of a promxy host -- since they all have prometheus compatible APIs.

I'm actually a little surprised no one has asked before, and even more surprised I don't have this in the FAQ -- so I have added it #227 .

glightfoot commented 4 years ago

Thank you!

jacksontj commented 4 years ago

You are very welcome, and as always if you have any other questions, issues, or suggestions let me know :)