oliver006
commented
2 years ago As for the timeout, looks to be the same issue as #579 - not sure what the solution is, but it seems to be a problem with AWS Elasticache clusters.
Redis_exporter is exceptionally slow - if you run the cluster with full debug mode and get a timestamp for each log line you should be able to see where the time is being spend. I agree, 60s is too long, esp if you don't run larger SCANs as part of the scrape.
As for the changing shard - is it possible that you're being connected to a different host of your cluster, each time the scrape is run? This could happen if you connect via a DNS name that round-robins between nodes. Again, could be a Elasticache-only isse, not sure.
ghost
Cluster monitoring questions and possible issues
Hi. I am running Redis 6 clusters on AWS with tls/ssl enabled. First of all i'd like to make sure that my assumptions are right. As for now i am trying to configure a single redis_exporter process per redis cluster, therefore i expect that exporter should be able to auto detect the cluster in a way as regular client does providing metrics with appropriate cardinality as well. That also makes sense if one concerns features that allow key count to be reported.
What version of redis_exporter are you running? v1.33.0-alpine
Running the exporter K8s deployment, default command line from vanilla dockerfile with following envs:
What i see is that exporter is somehow iterating all over 3 cluster shards (note db0 key count):
Some node metrics are also reported in log file f.e.:
There are some errors however:
Are those errors cricital? I am sure that this connection timeout is not caused by real network problems. There is a micorservice running on the same cluster and node that uses very same redis cluster. Logs and tracing shows that Redis is in use and i can cause new keys to be created. Connection and credentials should be fine since we get some metrics.
There is also another question. Redis_exporter is exceptionally slow.
curl localhost:8080/metricstakes more or less 60 seconds to complete. Once again, looks like cluster was correctly auto detected:But on the other side i can't get metrics for all shards at once. Each time i do curl i get values for a single shard:
Expected behavior It would be nice if anyone can answer my usage pattern questions first. If this exporter is considered as cluster-wide exporter then maybe this issue can be considered as a bug. In such case exporter should be a source of coherent full-cluster-state information each time we call GET /metrics.
Kind Regards Bart