search

OT-CONTAINER-KIT / helm-charts

A repository which that will contain helm charts with best and security practices.
https://ot-container-kit.github.io/helm-charts
49 stars 84 forks source link

Liveness/readiness probes fail when redisSecret is set #172

Closed diffuse closed 10 months ago

diffuse commented 10 months ago

Does this issue reproduce with the latest release? Yes, v0.15.11.

What operating system and processor architecture are you using (kubectl version)? minikube v1.30.1.

kubectl version Output
$ kubectl version
client: 1.27.3

What did you do? I configured the cluster with a secret defined, e.g.

redisCluster:
  clusterSize: 3
  clusterVersion: v7
  image: quay.io/opstree/redis
  tag: v7.0.12
  redisSecret:
    secretName: redis-secret
    secretKey: password
# ...snip...

What did you expect to see? The cluster leader and follower pods to come up successfully.

What did you see instead? The first redis cluster leader pod hangs at 0/1 Running, then transitions to CrashLoopBackOff. The output from describe on this pod shows that the liveness and readiness probes are failing because they can't authenticate:

Liveness probe failed: AUTH failed: DENIED Redis is running in protected mode because protected mode is enabled and no password is set for the default user.

...snip...
Readiness probe failed: AUTH failed: DENIED Redis is running in protected mode because protected mode is enabled and no password is set for the default user.

...snip...
diffuse commented 10 months ago

After further debugging, it turns out this was an issue with the values set in podSecurityContext and redisCluster.securityContext.

Everything works with the following values:

redisCluster:
   # ..snip...
   securityContext:
     runAsUser: 1000
     runAsGroup: 1000
     fsGroup: 1000
   # ..snip..
podSecurityContext:
  runAsUser: 1000
  runAsGroup: 1000
  fsGroup: 1000

Changing the values in either podSecurityContext OR redisCluster.securityContext to 1099 will reproduce the above issue. I'm not sure at which point in the setup this starts creating issues, but the end result was that the leader pod was running redis in protected mode with no password configured.