search

timescale / prometheus-postgresql-adapter

Use PostgreSQL as a remote storage database for Prometheus
Apache License 2.0
336 stars 66 forks source link

Leader election for Prometheus HA setup #37

Closed niksajakovljevic closed 6 years ago

niksajakovljevic commented 6 years ago

In HA setup two or more Prometheus instances scrape the same data and send it to adapters. We want to prevent multiple adapters to write the same/similar data (no consistency guarantees in HA Prometheus setup) in parallel. The idea is to use leader election and allow only one adapter (leader) to write to the database. Provided leader election implementation relies on PostgreSQL advisory locks and does not provide strong data guarantees (some duplicates or data loss possible during failover). REST interface for leader election should enable plugging in any external leader election system you might use already (eg. Zookeeper, Consul...)

RobAtticus commented 6 years ago

Let's start using the same standards for git commit messages as we do for main repo, so basically squash all the commits being merging.

Also maybe change the first line to Add support for leader election...?

niksajakovljevic commented 6 years ago

@amytai you can find an integration test here https://github.com/timescale/prometheus-postgresql-adapter/pull/37/files#diff-df609cea8c6beef224fb42b994af370bR122

niksajakovljevic commented 6 years ago

@erimatnor I fully understand your concerns. As we discussed the semantics of read.only is a bit different so I would keep it separate. I added additional Election implementation that provides REST interface so one can plug in any external leader election mechanism (eg. Zookeeper, Consul..)