search

samber / awesome-prometheus-alerts

🚨 Collection of Prometheus alerting rules
https://samber.github.io/awesome-prometheus-alerts/
Other
6.73k stars 1.02k forks source link

RedisTooManyMasters in a multi-cluster setup #419

Open exolab opened 6 months ago

exolab commented 6 months ago

Hi,

We are seeing this alert after deploying multiple Redis clusters in different namespaces. AFAICT Redis itself does not consider this setup a problem.

Does the alert not take this scenario into account, or are we mistaken in thinking we do not have a real problem?

Shouldn't the query be

count(redis_instance_info{role="master"}) by (namespace) > 1 or something similar that takes this scenario into account?

https://github.com/samber/awesome-prometheus-alerts/blame/826be5877ffc129ca3511cd25af87d67ea67fb48/_data/rules.yml#L795

samber commented 6 months ago

Hi @exolab and thanks for the report.

Do you have a better label than "namespace" for the cluster differentiation?

Do you know if redis_exporter exposes any cluster name?

exolab commented 6 months ago

Maybe by(service) would work. But I am by no means an expert on this.

DesireWithin commented 1 month ago

Hello @samber , I have same issue, my situation is setup a redis cluster with 3 master and 3 salve. Each master only handle part of slot:

图片 图片

This rule and RedisDisconnectedSlaves will calculated a wrong result.

How about using this formula to determine the cluster status? redis_cluster_state{%(redisExporterSelector)s} == 0