The goal is to create a monitoring setup where each server monitors itself when it comes failing systemd services, disk or RAM filling up, …. In addition each prometheus will monitor remote prometheus and alertmanager instances for signs of failure (e.g. being unreachable, errors in notification delivery, dropping alerts).
A lot of metrics (especially histograms from prometheus or alertmanager) are being dropped before ingestion to disk save on space and memory.
Depending on how many servers we may or may not have in the future this could probably use some kind of overhaul since we rightnow have n^2 monitoring peer relationships (not even speaking of possible duplicated alerts).
I hope the nix code is not to complex, my goal (I did possibly nerdsinpe myself a bit to hard), if there are any questions on that I'm more than happy to answer them (and try to simplify the code)!
The goal is to create a monitoring setup where each server monitors itself when it comes failing systemd services, disk or RAM filling up, …. In addition each prometheus will monitor remote prometheus and alertmanager instances for signs of failure (e.g. being unreachable, errors in notification delivery, dropping alerts).
A lot of metrics (especially histograms from prometheus or alertmanager) are being dropped before ingestion to disk save on space and memory.
Depending on how many servers we may or may not have in the future this could probably use some kind of overhaul since we rightnow have n^2 monitoring peer relationships (not even speaking of possible duplicated alerts).
I hope the nix code is not to complex, my goal (I did possibly nerdsinpe myself a bit to hard), if there are any questions on that I'm more than happy to answer them (and try to simplify the code)!