[bitnami/redis-sentinel] Redis Sentinel container liveness probe failures resulting in sentinel container restart

[karthicksndr]

Name and Version

bitnami/redis 18.18.0

What architecture are you using?

None

What steps will reproduce the bug?

1. Deploy Redis with Sentinel enabled and with 3 replicas with timeoutSeconds set to 15 for liveness and readiness probe.
2. Use a Java Spring app or any app of your choice to connect to Redis
3. This issue is intermittent and has no patterns.
4. So wait until the Sentinel container restart (it might hours or days)
5. If you could spot container restart, you will see a liveness probe failure
6. Exit code of the container at times could be 137, or Exit code 0.
7. You might think 137 error code is container resources limitations, or worker VM resource limitations, but it would have barely used CPU and memory.

Are you using any custom parameters or values?

Deploy the helm chart with sentinel enabled and setting the number of replicas to 3

replicaCount: 3
sentinel:
  enabled: true
  masterSet: my-master
  quorum: "2"
  livenessProbe:
    enabled: true
    initialDelaySeconds: 20
    periodSeconds: 10
    timeoutSeconds: 15
    successThreshold: 1
    failureThreshold: 5

What is the expected behavior?

No Sentinel container restarts with liveness probe failures.

What do you see instead?

Kubelet logs:

Memory usage: not even 1/10th of the memory limits (500 Mi) and 1/5th of memory requests (250 Mi)

CPU throttling:

Container resources at runtime:

Additional information

Outstanding question: Why the liveness probe fails

Sometimes with Exit code 137 Sometimes with Exit code 0 ( Purposely stopped ) with "ExecSync cmd from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"

What we have validated:

No resource issues in container (memory and CPU) No resource issues in other containers in the pod (redis, metrics, fluentbit) No resource issues in the node

[javsalgar]

Hi,

I see in the screenshot that it is adding more details like "command ..." but I cannot see it. Could you share more details on these kubelet logs?

[karthicksndr]

Thanks @javsalgar for looking into it.

k8s-node-vm-logs-redis-sentinel-restart.txt

Attached Kubelet logs. Note that the liveness probes were failing from 20:55 and the pod restarted around 20:57 after 5 liveness probe failures.

[javsalgar]

The only thing that comes to mind, because of this context deadline exceeded is some sort of connection issue due to the networking. I'm afraid that going further is a bit beyond of the support we can offer, but let's see if someone from the community could provide some insight on what could be happening.

If these issues are something transient, maybe you could try increasing the tolerance of the probes.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions[bot]]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

[karthicksndr]

Keeping this open for other engineers to comment.