[bitnami/redis-cluster] 6.3.9 fails if usePassword is set to false

[Oldervoll]

Which chart: redis-cluster 6.3.9

Describe the bug Creating a redis cluster without password fails. The liveness and readiness probe fails for containers during creation.

To Reproduce helm upgrade redis bitnami/redis-cluster --install --namespace redis --create-namespace --set fullnameOverride=redis --set usePassword=false

Expected behavior Redis containers liveness and readiness probe does not fail during startup.

Version of Helm and Kubernetes:

• Output of helm version:

version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}

• Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:45:37Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.9", GitCommit:"8dc42052754bfacb111a0426830122dd9cc9cfa0", GitTreeState:"clean", BuildDate:"2021-08-31T00:01:04Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}

Additional context Add any other context about the problem here.

From kubectl describe:

  Normal   Created                 11s (x2 over 38s)  kubelet                  Created container redis
  Normal   Started                 11s (x2 over 37s)  kubelet                  Started container redis
  Normal   Killing                 11s                kubelet                  Container redis failed liveness probe, will be restarted
  Warning  Unhealthy               11s                kubelet                  Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: container is in CONTAINER_EXITED state
  Warning  Unhealthy               1s (x7 over 31s)   kubelet                  Liveness probe failed:
AUTH failed: ERR AUTH <password> called without any password configured for the default user. Are you sure your configuration is correct?
  Warning  Unhealthy  1s (x6 over 31s)  kubelet  Readiness probe failed: AUTH failed: ERR AUTH <password> called without any password configured for the default user. Are you sure your configuration is correct?

[carrodher]

Unfortunately, I am not able to reproduce the issue:

$ kubectl create ns redis
namespace/redis created

$ helm install redis bitnami/redis-cluster --namespace redis --set fullnameOverride=redis --set usePassword=false
NAME: redis
LAST DEPLOYED: Wed Oct  6 08:54:53 2021
NAMESPACE: redis
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

You have deployed a Redis™ Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis™ cluster:

1. Run a Redis™ pod that you can use as a client:
kubectl run --namespace redis redis-client --rm --tty -i --restart='Never' \

--image docker.io/bitnami/redis-cluster:6.2.5-debian-10-r61 -- bash

2. Connect using the Redis™ CLI:

redis-cli -c -h redis

$ kubectl get pods -n redis
NAME      READY   STATUS    RESTARTS   AGE
redis-0   1/1     Running   0          4m27s
redis-1   1/1     Running   0          4m27s
redis-2   1/1     Running   0          4m27s
redis-3   1/1     Running   0          4m27s
redis-4   1/1     Running   0          4m27s
redis-5   1/1     Running   0          4m27s

Everything is up and running without any restart for some minutes and I can't see any issue.

Can you try in a new namespace using a new name? Take into account the PVCs are not removed with helm delete if you installed the chart previously using a password and reinstalled it again in the same namespace and/or with the same name but without using a password, there can be some discrepancies between the information stored in the PVC and the one from the new deployment.

[Oldervoll]

Hi @carrodher! Thanks for the fast reply.

There was no prior resources in the redis namespace before cluster creation. I tried now with a new namespace, and still getting the same issues. If I specify version 6.3.8 it works.

This is the description of one of the pods today:

Events:
  Type     Reason                  Age                     From                     Message
  ----     ------                  ----                    ----                     -------
  Warning  FailedScheduling        8m56s                   default-scheduler        0/3 nodes are available: 1 node(s) had taint {os: windows}, that the pod didn't tolerate, 2 node(s) exceed max volume count.
  Normal   Scheduled               7m7s                    default-scheduler        Successfully assigned redistest/redis-3 to aks-nodepool1-33819086-vmss000002
  Warning  FailedScheduling        8m56s                   default-scheduler        0/3 nodes are available: 1 node(s) had taint {os: windows}, that the pod didn't tolerate, 2 node(s) exceed max volume count.
  Normal   TriggeredScaleUp        8m51s                   cluster-autoscaler       pod triggered scale-up: [{aks-nodepool1-33819086-vmss 2->3 (max: 100)}]
  Normal   SuccessfulAttachVolume  6m47s                   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-4a95669e-624c-45fa-9e54-70686e8b032f"
  Normal   Pulling                 6m32s                   kubelet                  Pulling image "docker.io/bitnami/redis-cluster:6.2.6-debian-10-r0"
  Normal   Pulled                  6m27s                   kubelet                  Successfully pulled image "docker.io/bitnami/redis-cluster:6.2.6-debian-10-r0" in 5.267604242s
  Normal   Killing                 5m58s                   kubelet                  Container redis failed liveness probe, will be restarted
  Normal   Created                 5m28s (x2 over 6m25s)   kubelet                  Created container redis
  Normal   Pulled                  5m28s                   kubelet                  Container image "docker.io/bitnami/redis-cluster:6.2.6-debian-10-r0" already present on machine
  Normal   Started                 5m28s (x2 over 6m25s)   kubelet                  Started container redis
  Warning  Unhealthy               5m23s (x6 over 6m18s)   kubelet                  Liveness probe failed:
Could not connect to Redis at localhost:6379: Connection refused
  Warning  Unhealthy  89s (x6 over 2m44s)  kubelet  Readiness probe failed: AUTH failed: ERR AUTH <password> called without any password configured for the default user. Are you sure your configuration is correct?

This is the logs:

PS C:\360-docker\kubernetes> kubectl logs -n redistest redis-3
redis-cluster 10:18:51.33
redis-cluster 10:18:51.33 Welcome to the Bitnami redis-cluster container
redis-cluster 10:18:51.34 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-redis-cluster
redis-cluster 10:18:51.34 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-redis-cluster/issues
redis-cluster 10:18:51.34
redis-cluster 10:18:51.34 INFO  ==> ** Starting Redis setup **
redis-cluster 10:18:51.37 WARN  ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
redis-cluster 10:18:51.37 INFO  ==> Initializing Redis
redis-cluster 10:18:51.38 INFO  ==> Setting Redis config file
Changing old IP 10.240.0.96 by the new one 10.240.0.96
Changing old IP 10.240.0.10 by the new one 10.240.0.10
Changing old IP 10.240.0.116 by the new one 10.240.0.116
Changing old IP 10.240.0.110 by the new one 10.240.0.110
Changing old IP 10.240.0.103 by the new one 10.240.0.103
Changing old IP 10.240.0.120 by the new one 10.240.0.120
redis-cluster 10:18:51.49 INFO  ==> ** Redis setup finished! **

1:C 06 Oct 2021 10:18:51.545 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 06 Oct 2021 10:18:51.545 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 06 Oct 2021 10:18:51.545 # Configuration loaded
1:M 06 Oct 2021 10:18:51.546 * monotonic clock: POSIX clock_gettime
1:M 06 Oct 2021 10:18:51.548 * Node configuration loaded, I'm 9f278f958d7554c84d81408a676077a0f5f1dd25
                _._
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 6.2.6 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in cluster mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 1
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |           https://redis.io
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
              `-.__.-'

1:M 06 Oct 2021 10:18:51.549 # Server initialized
1:M 06 Oct 2021 10:18:51.549 * Ready to accept connections
1:signal-handler (1633515559) Received SIGTERM scheduling shutdown...
1:M 06 Oct 2021 10:19:19.438 # User requested shutdown...
1:M 06 Oct 2021 10:19:19.438 * Calling fsync() on the AOF file.
1:M 06 Oct 2021 10:19:19.438 * Saving the final RDB snapshot before exiting.
1:M 06 Oct 2021 10:19:19.444 * DB saved on disk
1:M 06 Oct 2021 10:19:19.444 * Removing the pid file.
1:M 06 Oct 2021 10:19:19.445 # Redis is now ready to exit, bye bye...

Let me know if I can provide more information that can be useful.

[carrodher]

Changes from 6.3.8 to 6.3.9 were done in this commit https://github.com/bitnami/charts/commit/a3226a87afacfae806eaa36ef6c498b772dccbb8, basically, the container image was bumped to a new Redis version:

-  tag: 6.2.5-debian-10-r61
+  tag: 6.2.6-debian-10-r0

Since I'm still not able to reproduce the issue, can you try the following workaround?

1. the new chart with the previous image

   helm install redis1 bitnami/redis-cluster --version 6.3.9 --namespace redis1 --create-namespace --set fullnameOverride=redis --set usePassword=false --set image.tag=6.2.5-debian-10-r61

2. the old chart with the new image

   helm install redis2 bitnami/redis-cluster --version 6.3.8 --namespace redis2 --create-namespace --set fullnameOverride=redis --set usePassword=false --set image.tag=6.2.6-debian-10-r0

[Oldervoll]

redis1                                 redis-0                                         1/1     Running                      0          171m
redis1                                 redis-1                                         1/1     Running                      1          172m
redis1                                 redis-2                                         1/1     Running                      2          3h8m
redis1                                 redis-3                                         1/1     Running                      0          3h8m
redis1                                 redis-4                                         1/1     Running                      1          172m
redis1                                 redis-5                                         1/1     Running                      1          172m
redis2                                 redis-0                                         0/1     CrashLoopBackOff             63         3h8m
redis2                                 redis-1                                         0/1     CrashLoopBackOff             63         3h8m
redis2                                 redis-2                                         0/1     CrashLoopBackOff             64         3h8m
redis2                                 redis-3                                         0/1     CrashLoopBackOff             64         3h8m
redis2                                 redis-4                                         0/1     CrashLoopBackOff             65         3h8m
redis2                                 redis-5                                         0/1     CrashLoopBackOff             63         3h8m

So it seems the issue is with the new image. Weird that this fails for me, but not for you. I'm using AKS btw.

[carrodher]

I was able to reproduce the issue:

## Current chart with and without `--set usePassword=false`
##
$ helm install redis bitnami/redis-cluster --namespace redis --create-namespace --set usePassword=false
$ helm install redis-no bitnami/redis-cluster --namespace redis-no --create-namespace

## Current chart and old image with and without `--set usePassword=false`
##
$ helm install redis1 bitnami/redis-cluster --version 6.3.9 --namespace redis1 --create-namespace --set usePassword=false --set image.tag=6.2.5-debian-10-r61
$ helm install redis1-no bitnami/redis-cluster --version 6.3.9 --namespace redis1-no --create-namespace --set image.tag=6.2.5-debian-10-r61

## Old chart and current image with and without `--set usePassword=false`
##
$ helm install redis2 bitnami/redis-cluster --version 6.3.8 --namespace redis2 --create-namespace --set usePassword=false --set image.tag=6.2.6-debian-10-r0
$ helm install redis2-no bitnami/redis-cluster --version 6.3.8 --namespace redis2-no --create-namespace --set image.tag=6.2.6-debian-10-r0

## When `--set usePassword=false` is not used, always work
##
$ kubectl get pods -n redis-no
NAME                       READY   STATUS    RESTARTS   AGE
redis-no-redis-cluster-0   1/1     Running   0          4m51s
redis-no-redis-cluster-1   1/1     Running   0          4m51s
redis-no-redis-cluster-2   1/1     Running   0          4m51s
redis-no-redis-cluster-3   1/1     Running   0          4m50s
redis-no-redis-cluster-4   1/1     Running   0          4m50s
redis-no-redis-cluster-5   1/1     Running   0          4m50s

$ kubectl get pods -n redis1-no
NAME                        READY   STATUS    RESTARTS   AGE
redis1-no-redis-cluster-0   1/1     Running   0          4m6s
redis1-no-redis-cluster-1   1/1     Running   0          4m6s
redis1-no-redis-cluster-2   1/1     Running   0          4m6s
redis1-no-redis-cluster-3   1/1     Running   0          4m6s
redis1-no-redis-cluster-4   1/1     Running   0          4m6s
redis1-no-redis-cluster-5   1/1     Running   0          4m6s

$ kubectl get pods -n redis2-no
NAME                        READY   STATUS    RESTARTS   AGE
redis2-no-redis-cluster-0   1/1     Running   0          3m23s
redis2-no-redis-cluster-1   1/1     Running   0          3m23s
redis2-no-redis-cluster-2   1/1     Running   0          3m23s
redis2-no-redis-cluster-3   1/1     Running   0          3m23s
redis2-no-redis-cluster-4   1/1     Running   0          3m23s
redis2-no-redis-cluster-5   1/1     Running   0          3m23s

## When `--set usePassword=false` is used it doesn't work with the new image
##
$ kubectl get pods -n redis
NAME                    READY   STATUS             RESTARTS   AGE
redis-redis-cluster-0   0/1     CrashLoopBackOff   5          5m35s
redis-redis-cluster-1   0/1     CrashLoopBackOff   6          5m35s
redis-redis-cluster-2   0/1     Running            6          5m35s
redis-redis-cluster-3   0/1     Running            6          5m35s
redis-redis-cluster-4   0/1     CrashLoopBackOff   5          5m35s
redis-redis-cluster-5   0/1     CrashLoopBackOff   6          5m35s

$ kubectl get pods -n redis1
NAME                     READY   STATUS    RESTARTS   AGE
redis1-redis-cluster-0   1/1     Running   0          4m37s
redis1-redis-cluster-1   1/1     Running   0          4m37s
redis1-redis-cluster-2   1/1     Running   0          4m37s
redis1-redis-cluster-3   1/1     Running   0          4m37s
redis1-redis-cluster-4   1/1     Running   0          4m37s
redis1-redis-cluster-5   1/1     Running   0          4m37s

$ kubectl get pods -n redis2
NAME                     READY   STATUS             RESTARTS   AGE
redis2-redis-cluster-0   0/1     Running            5          3m56s
redis2-redis-cluster-1   0/1     Running            5          3m56s
redis2-redis-cluster-2   0/1     CrashLoopBackOff   5          3m56s
redis2-redis-cluster-3   0/1     Running            5          3m56s
redis2-redis-cluster-4   0/1     CrashLoopBackOff   5          3m56s
redis2-redis-cluster-5   0/1     Running            5          3m56s

So yes, we can confirm the issue appears when using an image from the new version (6.2.6) and usePassword is set to false. Taking a look at the changes in that version there is not any change on our side (https://github.com/bitnami/bitnami-docker-redis-cluster/commit/c6bcb2d4e5a544e56a86f0ad5e90e0c0df1bc0f2) apart from bumping the version with the upstream source code. I was taking a look at the upstream release notes but I can't see anything relevant related to auth/passwords, see https://raw.githubusercontent.com/redis/redis/6.2/00-RELEASENOTES

usePassword is used to set some env. variables for the authentication:

$ helm template redis bitnami/redis-cluster --namespace redis --create-namespace --set usePassword=false -s templates/redis-statefulset.yaml > false.txt
$ helm template redis bitnami/redis-cluster --namespace redis --create-namespace --set usePassword=true -s templates/redis-statefulset.yaml > true.txt
$ colordiff false.txt true.txt

29c29
-         checksum/secret: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
+         checksum/secret: a90c30125d9ac37127175c1f959fcccb5fa55c2ae341844aa865af3d67a9a6bd
82,83c82,91
-             - name: ALLOW_EMPTY_PASSWORD
-               value: "yes"
+             - name: REDISCLI_AUTH
+               valueFrom:
+                 secretKeyRef:
+                   name: redis-redis-cluster
+                   key: redis-password
+             - name: REDIS_PASSWORD
+               valueFrom:
+                 secretKeyRef:
+                   name: redis-redis-cluster
+                   key: redis-password

Liveness probes are failing with the following error (when ALLOW_EMPTY_PASSWORD is yes):

$ kubectl describe pod redis-redis-cluster-0 -n redis
...
    Environment:
      POD_NAME:              redis-redis-cluster-0 (v1:metadata.name)
      REDIS_NODES:           redis-redis-cluster-0.redis-redis-cluster-headless redis-redis-cluster-1.redis-redis-cluster-headless redis-redis-cluster-2.redis-redis-cluster-headless redis-redis-cluster-3.redis-redis-cluster-headless redis-redis-cluster-4.redis-redis-cluster-headless redis-redis-cluster-5.redis-redis-cluster-headless
      ALLOW_EMPTY_PASSWORD:  yes
      REDIS_AOF_ENABLED:     yes
      REDIS_TLS_ENABLED:     no
      REDIS_PORT:            6379
...
  Warning  Unhealthy  19m (x49 over 39m)  kubelet, gke-dev-default-pool-ab651c88-hzxw  Liveness probe failed:
AUTH failed: ERR AUTH <password> called without any password configured for the default user. Are you sure your configuration is correct?
  Warning  BackOff  4m13s (x138 over 36m)  kubelet, gke-dev-default-pool-ab651c88-hzxw  Back-off restarting failed container

I just created an internal task to properly investigate the issue.

[carrodher]

Another user created this PR (https://github.com/bitnami/charts/pull/7771) trying to solve this issue but it seems it was not fully solved. I am still working on a solution in the container image itself

[Oldervoll]

@carrodher looks like this works again with image docker.io/bitnami/redis-cluster:6.2.6-debian-10-r0 and chart version redis-cluster-7.0.7.

Has any fixes been applied by purpose? If so, I guess we can close this issue.

[carrodher]

The image was not modified, as we are using immutable tags, 6.2.6-debian-10-r0 will never be modified; at some point, the chart will be updated to use a different tag, for example, the latest one at this moment is 6.2.6-debian-10-r20. In this case, the chart is using the same image since the app version was bumped from 6.2.5 to 6.2.6, see https://github.com/bitnami/charts/commit/a3226a87afacfae806eaa36ef6c498b772dccbb8

Regarding changes in the chart, yes, taking a look at the commits history there are several changes that were done to improve the chart, being the following ones the most relevant:

• https://github.com/bitnami/charts/pull/7710, this was a major change since the whole chart was refactored
• https://github.com/bitnami/charts/pull/7771, this is related to this issue, although in my initial tests the issue appeared, probably I was using a previous version without taking into account those changes

I tried the different scenarios and everything is working fine, so we can consider this issue as fixed:

$ helm install redis-no bitnami/redis-cluster --namespace redis-no --create-namespace
NAME: redis-no
LAST DEPLOYED: Tue Oct 26 13:36:00 2021
NAMESPACE: redis-no
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis-cluster
CHART VERSION: 7.0.7
APP VERSION: 6.2.6

** Please be patient while the chart is being deployed **

To get your password run:
    export REDIS_PASSWORD=$(kubectl get secret --namespace "redis-no" redis-no-redis-cluster -o jsonpath="{.data.redis-password}" | base64 --decode)

You have deployed a Redis™ Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis™ cluster:

1. Run a Redis™ pod that you can use as a client:
kubectl run --namespace redis-no redis-no-redis-cluster-client --rm --tty -i --restart='Never' \
 --env REDIS_PASSWORD=$REDIS_PASSWORD \
--image docker.io/bitnami/redis-cluster:6.2.6-debian-10-r0 -- bash

2. Connect using the Redis™ CLI:

redis-cli -c -h redis-no-redis-cluster -a $REDIS_PASSWORD

$ helm install redis bitnami/redis-cluster --namespace redis --create-namespace --set usePassword=false
NAME: redis
LAST DEPLOYED: Tue Oct 26 13:36:09 2021
NAMESPACE: redis
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis-cluster
CHART VERSION: 7.0.7
APP VERSION: 6.2.6

** Please be patient while the chart is being deployed **

You have deployed a Redis™ Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis™ cluster:

1. Run a Redis™ pod that you can use as a client:
kubectl run --namespace redis redis-redis-cluster-client --rm --tty -i --restart='Never' \

--image docker.io/bitnami/redis-cluster:6.2.6-debian-10-r0 -- bash

2. Connect using the Redis™ CLI:

redis-cli -c -h redis-redis-cluster

$ kubectl get pods -n redis
NAME                    READY   STATUS    RESTARTS   AGE
redis-redis-cluster-0   1/1     Running   0          25m
redis-redis-cluster-1   1/1     Running   0          25m
redis-redis-cluster-2   1/1     Running   0          25m
redis-redis-cluster-3   1/1     Running   0          25m
redis-redis-cluster-4   1/1     Running   0          25m
redis-redis-cluster-5   1/1     Running   0          25m

$ kubectl get pods -n redis-no
NAME                       READY   STATUS    RESTARTS   AGE
redis-no-redis-cluster-0   1/1     Running   0          25m
redis-no-redis-cluster-1   1/1     Running   0          25m
redis-no-redis-cluster-2   1/1     Running   0          25m
redis-no-redis-cluster-3   1/1     Running   0          25m
redis-no-redis-cluster-4   1/1     Running   0          25m
redis-no-redis-cluster-5   1/1     Running   0          25m