[HELP] Monitoring Stack on Kubernetes

[MiguelNdeCarvalho]

Hey, I am mounting a Kubernetes stack to monitor with High Avaibility. This stack will contain:

• 2 x Prometheus
• 2 x Grafana
• 2 x AlertManager
• 2 x BlackBox Exporter

I want to use Prometheus with TimeScaleDB to save my metrics. So what should I do ? Run 2 adapters one for each prometheus intance? How can it select the leader ? Thanks, Miguel de Carvalho

[virasatkhan]

I can try to answer your question on how to select leader.

When you run your pg_adapter that eventually writes into your DB, Start both your adapters with below lines.

-leader-election.pg-advisory-lock-id=1 -leader-election.pg-advisory-lock.prometheus-timeout=6m

the lock-id must be same on the 2nd pg_adapter that you are going to start and the timeout is something that you may want to tweak based on your scraping interval.

This will ensure the other pg_adapter takes the lock when one goes down.

Basically.

Your Prometheus instance (P1) is reading/writing to a pg_adapter (PG1) and Another Prometheus instance (P2) is reading/writing to another pg_adapter( PG2)

Both PG1 and PG2 are writing to a single DB with that lock configured. You can check your container logs where one adapter will write and other will say some message that means it did not acquire log and is in passive state.

[MiguelNdeCarvalho]

Really thanks for your help :+1:

[niksajakovljevic]

@MiguelNdeCarvalho You might find this useful https://timescale.ghost.io/blog/prometheus-ha-postgresql-8de68d19b6f5/