Recent outage surfaced that when promscale is down, our whole monitoring system can go down due to misconfigured remote_read. Before change from this PR prometheus ALWAYS wanted to get data from promscale when querying data or computing recording rules. This in turn created a single point of failure even in a setup with prometheus in HA. This PR is fixing this major issue and allows prometheus to be used as a buffer when promscale is down.
Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)
This should allow retreiving data from prometheus even when promscale is down and thus increasing overal reliability of the stack.
Signed-off-by: Paweł Krupa (paulfantom) pawel@krupa.net.pl
What this PR does / why we need it
Recent outage surfaced that when promscale is down, our whole monitoring system can go down due to misconfigured remote_read. Before change from this PR prometheus ALWAYS wanted to get data from promscale when querying data or computing recording rules. This in turn created a single point of failure even in a setup with prometheus in HA. This PR is fixing this major issue and allows prometheus to be used as a buffer when promscale is down.
Which issue this PR fixes
(optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close that issue when PR gets merged)Special notes for your reviewer
Checklist