-
We recently enabled some global recording rules to generate cross-dc metrics, and are noticing some inconsistencies with the values written to the remote-write-exporter. I'll try to illustrate the iss…
-
Currently we determine the phase of a target's scrape by it's labels and params. This means that a HA pair will both try to scrape targets at exactly the same time, causing a bigger load spike and pot…
-
We should be advising people to run their Prometheus instance in HA, so they don't loose data if the machine Prometheus is running on fails. However, in a nieve setup, this will lead to a high percen…
-
I have a 3 node rabbitmq-ha cluster running on development aks cluster. At night we power down the VM's. When the VMS come back online my user credentials no longer work, even though the user still ex…
-
Hello TSDB folks!
We are chasing for a safe solution to quickly "terminate" Prometheus server without losing any monitoring data stored in memory (and WAL). By terminate, we mean killing whole inst…
-
Hi ,
I am using aks with a rabbitmq-ha , the cluster using the mirror queue .
And after some time - I am getting this warning and the pod state is crash loop -
The log is
>
warning: /var/…
-
### The issue
It looks like Promxy fetches all the raw data points for the given time range in the query from upstream nodes and then merges them. This may take big amounts of network bandwidth, CP…
-
Prometheus and Kubernetes have common ground in their usage of labels, which makes them a very good fit for each other. However, Kubernetes has a more flexible pattern for label names. Prometheus only…
fabxc updated
5 years ago
-
Store nodes are currently generally run as a single replica. It's not super critical to have HA in general since several hours or even days of recent data are HA via the Prometheus servers. But for so…
fabxc updated
5 years ago
-
RUNBOOK was added in a hurry in #843 for an internal demo of one of our users, which didn't give it enough time to be fully discussed. The demo has been done, so we can reconsider this.
I think we s…