search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
22.75k stars 3.31k forks source link

Unreachable Redis Sentinel in Loki #11564

Open aryalireza opened 6 months ago

aryalireza commented 6 months ago

I got this error when I configured Loki to use Redis Sentinel nodes and in Loki logs, I got these messages and errors:

level=warn ts=2023-12-31T17:46:51.264117495Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - embedded-cache"
level=warn ts=2023-12-31T17:46:51.26450088Z caller=experimental.go:20 msg="experimental feature in use" feature="Redis cache - chunksredis"

redis: 2023/12/31 17:46:51 sentinel.go:514: sentinel: GetMasterAddrByName master="mymaster" failed: context deadline exceeded
redis: 2023/12/31 17:46:51 sentinel.go:514: sentinel: GetMasterAddrByName master="mymaster" failed: context deadline exceeded
redis: 2023/12/31 17:46:51 sentinel.go:514: sentinel: GetMasterAddrByName master="mymaster" failed: context deadline exceeded

level=error ts=2023-12-31T17:46:51.765505306Z caller=redis_cache.go:32 msg=" error connecting to redis" name=chunksredis err="redis: all sentinels specified in the configuration are unreachable"

I thought maybe I had some network access issues or misconfiguration in my Redis Sentinelsm but when I use redis-cli and this command = > keys * , I can see Loki microservices are using Redis :))) But still I can see these errors in ingester log:

level=error ts=2023-12-31T17:54:59.280142198Z caller=redis_cache.go:61 msg="failed to put to redis" name=chunksredis err="read tcp 10.42.44.222:37908->192.168.175.164:6379: i/o timeout"

I am using Loki microservice/distributed mode that I deployed on a K8S cluster via helm chart version: Loki-distributed-0.77.0 and app version:2.9.2.

JStickler commented 5 months ago

Unless you think this is a bug, questions have a better chance of being answered if you ask them on the community forums.

haalb commented 3 months ago

@aryalireza I have the same error. Did you figure out why this error occurs and a solution for it?

aryalireza commented 3 months ago

@haalb not yet! I also changed the version of Redis and nothing changed. I don't know what is going on when this issue is raised.

aryalireza commented 3 months ago

Also asked in Grafana Lab's forum: https://community.grafana.com/t/unreachable-redis-sentinel-in-loki/116030