Closed Oldervoll closed 2 years ago
Unfortunately, I am not able to reproduce the issue:
$ kubectl create ns redis
namespace/redis created
$ helm install redis bitnami/redis-cluster --namespace redis --set fullnameOverride=redis --set usePassword=false
NAME: redis
LAST DEPLOYED: Wed Oct 6 08:54:53 2021
NAMESPACE: redis
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **
You have deployed a Redis™ Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis™ cluster:
1. Run a Redis™ pod that you can use as a client:
kubectl run --namespace redis redis-client --rm --tty -i --restart='Never' \
--image docker.io/bitnami/redis-cluster:6.2.5-debian-10-r61 -- bash
2. Connect using the Redis™ CLI:
redis-cli -c -h redis
$ kubectl get pods -n redis
NAME READY STATUS RESTARTS AGE
redis-0 1/1 Running 0 4m27s
redis-1 1/1 Running 0 4m27s
redis-2 1/1 Running 0 4m27s
redis-3 1/1 Running 0 4m27s
redis-4 1/1 Running 0 4m27s
redis-5 1/1 Running 0 4m27s
Everything is up and running without any restart for some minutes and I can't see any issue.
Can you try in a new namespace using a new name? Take into account the PVCs are not removed with helm delete
if you installed the chart previously using a password and reinstalled it again in the same namespace and/or with the same name but without using a password, there can be some discrepancies between the information stored in the PVC and the one from the new deployment.
Hi @carrodher! Thanks for the fast reply.
There was no prior resources in the redis namespace before cluster creation. I tried now with a new namespace, and still getting the same issues. If I specify version 6.3.8 it works.
This is the description of one of the pods today:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 8m56s default-scheduler 0/3 nodes are available: 1 node(s) had taint {os: windows}, that the pod didn't tolerate, 2 node(s) exceed max volume count.
Normal Scheduled 7m7s default-scheduler Successfully assigned redistest/redis-3 to aks-nodepool1-33819086-vmss000002
Warning FailedScheduling 8m56s default-scheduler 0/3 nodes are available: 1 node(s) had taint {os: windows}, that the pod didn't tolerate, 2 node(s) exceed max volume count.
Normal TriggeredScaleUp 8m51s cluster-autoscaler pod triggered scale-up: [{aks-nodepool1-33819086-vmss 2->3 (max: 100)}]
Normal SuccessfulAttachVolume 6m47s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-4a95669e-624c-45fa-9e54-70686e8b032f"
Normal Pulling 6m32s kubelet Pulling image "docker.io/bitnami/redis-cluster:6.2.6-debian-10-r0"
Normal Pulled 6m27s kubelet Successfully pulled image "docker.io/bitnami/redis-cluster:6.2.6-debian-10-r0" in 5.267604242s
Normal Killing 5m58s kubelet Container redis failed liveness probe, will be restarted
Normal Created 5m28s (x2 over 6m25s) kubelet Created container redis
Normal Pulled 5m28s kubelet Container image "docker.io/bitnami/redis-cluster:6.2.6-debian-10-r0" already present on machine
Normal Started 5m28s (x2 over 6m25s) kubelet Started container redis
Warning Unhealthy 5m23s (x6 over 6m18s) kubelet Liveness probe failed:
Could not connect to Redis at localhost:6379: Connection refused
Warning Unhealthy 89s (x6 over 2m44s) kubelet Readiness probe failed: AUTH failed: ERR AUTH <password> called without any password configured for the default user. Are you sure your configuration is correct?
This is the logs:
PS C:\360-docker\kubernetes> kubectl logs -n redistest redis-3
redis-cluster 10:18:51.33
redis-cluster 10:18:51.33 Welcome to the Bitnami redis-cluster container
redis-cluster 10:18:51.34 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-redis-cluster
redis-cluster 10:18:51.34 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-redis-cluster/issues
redis-cluster 10:18:51.34
redis-cluster 10:18:51.34 INFO ==> ** Starting Redis setup **
redis-cluster 10:18:51.37 WARN ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
redis-cluster 10:18:51.37 INFO ==> Initializing Redis
redis-cluster 10:18:51.38 INFO ==> Setting Redis config file
Changing old IP 10.240.0.96 by the new one 10.240.0.96
Changing old IP 10.240.0.10 by the new one 10.240.0.10
Changing old IP 10.240.0.116 by the new one 10.240.0.116
Changing old IP 10.240.0.110 by the new one 10.240.0.110
Changing old IP 10.240.0.103 by the new one 10.240.0.103
Changing old IP 10.240.0.120 by the new one 10.240.0.120
redis-cluster 10:18:51.49 INFO ==> ** Redis setup finished! **
1:C 06 Oct 2021 10:18:51.545 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 06 Oct 2021 10:18:51.545 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 06 Oct 2021 10:18:51.545 # Configuration loaded
1:M 06 Oct 2021 10:18:51.546 * monotonic clock: POSIX clock_gettime
1:M 06 Oct 2021 10:18:51.548 * Node configuration loaded, I'm 9f278f958d7554c84d81408a676077a0f5f1dd25
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 6.2.6 (00000000/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in cluster mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6379
| `-._ `._ / _.-' | PID: 1
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | https://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
1:M 06 Oct 2021 10:18:51.549 # Server initialized
1:M 06 Oct 2021 10:18:51.549 * Ready to accept connections
1:signal-handler (1633515559) Received SIGTERM scheduling shutdown...
1:M 06 Oct 2021 10:19:19.438 # User requested shutdown...
1:M 06 Oct 2021 10:19:19.438 * Calling fsync() on the AOF file.
1:M 06 Oct 2021 10:19:19.438 * Saving the final RDB snapshot before exiting.
1:M 06 Oct 2021 10:19:19.444 * DB saved on disk
1:M 06 Oct 2021 10:19:19.444 * Removing the pid file.
1:M 06 Oct 2021 10:19:19.445 # Redis is now ready to exit, bye bye...
Let me know if I can provide more information that can be useful.
Changes from 6.3.8 to 6.3.9 were done in this commit https://github.com/bitnami/charts/commit/a3226a87afacfae806eaa36ef6c498b772dccbb8, basically, the container image was bumped to a new Redis version:
- tag: 6.2.5-debian-10-r61
+ tag: 6.2.6-debian-10-r0
Since I'm still not able to reproduce the issue, can you try the following workaround?
the new chart with the previous image
helm install redis1 bitnami/redis-cluster --version 6.3.9 --namespace redis1 --create-namespace --set fullnameOverride=redis --set usePassword=false --set image.tag=6.2.5-debian-10-r61
the old chart with the new image
helm install redis2 bitnami/redis-cluster --version 6.3.8 --namespace redis2 --create-namespace --set fullnameOverride=redis --set usePassword=false --set image.tag=6.2.6-debian-10-r0
redis1 redis-0 1/1 Running 0 171m
redis1 redis-1 1/1 Running 1 172m
redis1 redis-2 1/1 Running 2 3h8m
redis1 redis-3 1/1 Running 0 3h8m
redis1 redis-4 1/1 Running 1 172m
redis1 redis-5 1/1 Running 1 172m
redis2 redis-0 0/1 CrashLoopBackOff 63 3h8m
redis2 redis-1 0/1 CrashLoopBackOff 63 3h8m
redis2 redis-2 0/1 CrashLoopBackOff 64 3h8m
redis2 redis-3 0/1 CrashLoopBackOff 64 3h8m
redis2 redis-4 0/1 CrashLoopBackOff 65 3h8m
redis2 redis-5 0/1 CrashLoopBackOff 63 3h8m
So it seems the issue is with the new image. Weird that this fails for me, but not for you. I'm using AKS btw.
I was able to reproduce the issue:
## Current chart with and without `--set usePassword=false`
##
$ helm install redis bitnami/redis-cluster --namespace redis --create-namespace --set usePassword=false
$ helm install redis-no bitnami/redis-cluster --namespace redis-no --create-namespace
## Current chart and old image with and without `--set usePassword=false`
##
$ helm install redis1 bitnami/redis-cluster --version 6.3.9 --namespace redis1 --create-namespace --set usePassword=false --set image.tag=6.2.5-debian-10-r61
$ helm install redis1-no bitnami/redis-cluster --version 6.3.9 --namespace redis1-no --create-namespace --set image.tag=6.2.5-debian-10-r61
## Old chart and current image with and without `--set usePassword=false`
##
$ helm install redis2 bitnami/redis-cluster --version 6.3.8 --namespace redis2 --create-namespace --set usePassword=false --set image.tag=6.2.6-debian-10-r0
$ helm install redis2-no bitnami/redis-cluster --version 6.3.8 --namespace redis2-no --create-namespace --set image.tag=6.2.6-debian-10-r0
## When `--set usePassword=false` is not used, always work
##
$ kubectl get pods -n redis-no
NAME READY STATUS RESTARTS AGE
redis-no-redis-cluster-0 1/1 Running 0 4m51s
redis-no-redis-cluster-1 1/1 Running 0 4m51s
redis-no-redis-cluster-2 1/1 Running 0 4m51s
redis-no-redis-cluster-3 1/1 Running 0 4m50s
redis-no-redis-cluster-4 1/1 Running 0 4m50s
redis-no-redis-cluster-5 1/1 Running 0 4m50s
$ kubectl get pods -n redis1-no
NAME READY STATUS RESTARTS AGE
redis1-no-redis-cluster-0 1/1 Running 0 4m6s
redis1-no-redis-cluster-1 1/1 Running 0 4m6s
redis1-no-redis-cluster-2 1/1 Running 0 4m6s
redis1-no-redis-cluster-3 1/1 Running 0 4m6s
redis1-no-redis-cluster-4 1/1 Running 0 4m6s
redis1-no-redis-cluster-5 1/1 Running 0 4m6s
$ kubectl get pods -n redis2-no
NAME READY STATUS RESTARTS AGE
redis2-no-redis-cluster-0 1/1 Running 0 3m23s
redis2-no-redis-cluster-1 1/1 Running 0 3m23s
redis2-no-redis-cluster-2 1/1 Running 0 3m23s
redis2-no-redis-cluster-3 1/1 Running 0 3m23s
redis2-no-redis-cluster-4 1/1 Running 0 3m23s
redis2-no-redis-cluster-5 1/1 Running 0 3m23s
## When `--set usePassword=false` is used it doesn't work with the new image
##
$ kubectl get pods -n redis
NAME READY STATUS RESTARTS AGE
redis-redis-cluster-0 0/1 CrashLoopBackOff 5 5m35s
redis-redis-cluster-1 0/1 CrashLoopBackOff 6 5m35s
redis-redis-cluster-2 0/1 Running 6 5m35s
redis-redis-cluster-3 0/1 Running 6 5m35s
redis-redis-cluster-4 0/1 CrashLoopBackOff 5 5m35s
redis-redis-cluster-5 0/1 CrashLoopBackOff 6 5m35s
$ kubectl get pods -n redis1
NAME READY STATUS RESTARTS AGE
redis1-redis-cluster-0 1/1 Running 0 4m37s
redis1-redis-cluster-1 1/1 Running 0 4m37s
redis1-redis-cluster-2 1/1 Running 0 4m37s
redis1-redis-cluster-3 1/1 Running 0 4m37s
redis1-redis-cluster-4 1/1 Running 0 4m37s
redis1-redis-cluster-5 1/1 Running 0 4m37s
$ kubectl get pods -n redis2
NAME READY STATUS RESTARTS AGE
redis2-redis-cluster-0 0/1 Running 5 3m56s
redis2-redis-cluster-1 0/1 Running 5 3m56s
redis2-redis-cluster-2 0/1 CrashLoopBackOff 5 3m56s
redis2-redis-cluster-3 0/1 Running 5 3m56s
redis2-redis-cluster-4 0/1 CrashLoopBackOff 5 3m56s
redis2-redis-cluster-5 0/1 Running 5 3m56s
So yes, we can confirm the issue appears when using an image from the new version (6.2.6) and usePassword
is set to false. Taking a look at the changes in that version there is not any change on our side (https://github.com/bitnami/bitnami-docker-redis-cluster/commit/c6bcb2d4e5a544e56a86f0ad5e90e0c0df1bc0f2) apart from bumping the version with the upstream source code.
I was taking a look at the upstream release notes but I can't see anything relevant related to auth/passwords, see https://raw.githubusercontent.com/redis/redis/6.2/00-RELEASENOTES
usePassword
is used to set some env. variables for the authentication:
$ helm template redis bitnami/redis-cluster --namespace redis --create-namespace --set usePassword=false -s templates/redis-statefulset.yaml > false.txt
$ helm template redis bitnami/redis-cluster --namespace redis --create-namespace --set usePassword=true -s templates/redis-statefulset.yaml > true.txt
$ colordiff false.txt true.txt
29c29
- checksum/secret: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
+ checksum/secret: a90c30125d9ac37127175c1f959fcccb5fa55c2ae341844aa865af3d67a9a6bd
82,83c82,91
- - name: ALLOW_EMPTY_PASSWORD
- value: "yes"
+ - name: REDISCLI_AUTH
+ valueFrom:
+ secretKeyRef:
+ name: redis-redis-cluster
+ key: redis-password
+ - name: REDIS_PASSWORD
+ valueFrom:
+ secretKeyRef:
+ name: redis-redis-cluster
+ key: redis-password
Liveness probes are failing with the following error (when ALLOW_EMPTY_PASSWORD
is yes):
$ kubectl describe pod redis-redis-cluster-0 -n redis
...
Environment:
POD_NAME: redis-redis-cluster-0 (v1:metadata.name)
REDIS_NODES: redis-redis-cluster-0.redis-redis-cluster-headless redis-redis-cluster-1.redis-redis-cluster-headless redis-redis-cluster-2.redis-redis-cluster-headless redis-redis-cluster-3.redis-redis-cluster-headless redis-redis-cluster-4.redis-redis-cluster-headless redis-redis-cluster-5.redis-redis-cluster-headless
ALLOW_EMPTY_PASSWORD: yes
REDIS_AOF_ENABLED: yes
REDIS_TLS_ENABLED: no
REDIS_PORT: 6379
...
Warning Unhealthy 19m (x49 over 39m) kubelet, gke-dev-default-pool-ab651c88-hzxw Liveness probe failed:
AUTH failed: ERR AUTH <password> called without any password configured for the default user. Are you sure your configuration is correct?
Warning BackOff 4m13s (x138 over 36m) kubelet, gke-dev-default-pool-ab651c88-hzxw Back-off restarting failed container
I just created an internal task to properly investigate the issue.
Another user created this PR (https://github.com/bitnami/charts/pull/7771) trying to solve this issue but it seems it was not fully solved. I am still working on a solution in the container image itself
@carrodher looks like this works again with image docker.io/bitnami/redis-cluster:6.2.6-debian-10-r0 and chart version redis-cluster-7.0.7.
Has any fixes been applied by purpose? If so, I guess we can close this issue.
The image was not modified, as we are using immutable tags, 6.2.6-debian-10-r0
will never be modified; at some point, the chart will be updated to use a different tag, for example, the latest one at this moment is 6.2.6-debian-10-r20
. In this case, the chart is using the same image since the app version was bumped from 6.2.5
to 6.2.6
, see https://github.com/bitnami/charts/commit/a3226a87afacfae806eaa36ef6c498b772dccbb8
Regarding changes in the chart, yes, taking a look at the commits history there are several changes that were done to improve the chart, being the following ones the most relevant:
I tried the different scenarios and everything is working fine, so we can consider this issue as fixed:
$ helm install redis-no bitnami/redis-cluster --namespace redis-no --create-namespace
NAME: redis-no
LAST DEPLOYED: Tue Oct 26 13:36:00 2021
NAMESPACE: redis-no
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis-cluster
CHART VERSION: 7.0.7
APP VERSION: 6.2.6
** Please be patient while the chart is being deployed **
To get your password run:
export REDIS_PASSWORD=$(kubectl get secret --namespace "redis-no" redis-no-redis-cluster -o jsonpath="{.data.redis-password}" | base64 --decode)
You have deployed a Redis™ Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis™ cluster:
1. Run a Redis™ pod that you can use as a client:
kubectl run --namespace redis-no redis-no-redis-cluster-client --rm --tty -i --restart='Never' \
--env REDIS_PASSWORD=$REDIS_PASSWORD \
--image docker.io/bitnami/redis-cluster:6.2.6-debian-10-r0 -- bash
2. Connect using the Redis™ CLI:
redis-cli -c -h redis-no-redis-cluster -a $REDIS_PASSWORD
$ helm install redis bitnami/redis-cluster --namespace redis --create-namespace --set usePassword=false
NAME: redis
LAST DEPLOYED: Tue Oct 26 13:36:09 2021
NAMESPACE: redis
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis-cluster
CHART VERSION: 7.0.7
APP VERSION: 6.2.6
** Please be patient while the chart is being deployed **
You have deployed a Redis™ Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis™ cluster:
1. Run a Redis™ pod that you can use as a client:
kubectl run --namespace redis redis-redis-cluster-client --rm --tty -i --restart='Never' \
--image docker.io/bitnami/redis-cluster:6.2.6-debian-10-r0 -- bash
2. Connect using the Redis™ CLI:
redis-cli -c -h redis-redis-cluster
$ kubectl get pods -n redis
NAME READY STATUS RESTARTS AGE
redis-redis-cluster-0 1/1 Running 0 25m
redis-redis-cluster-1 1/1 Running 0 25m
redis-redis-cluster-2 1/1 Running 0 25m
redis-redis-cluster-3 1/1 Running 0 25m
redis-redis-cluster-4 1/1 Running 0 25m
redis-redis-cluster-5 1/1 Running 0 25m
$ kubectl get pods -n redis-no
NAME READY STATUS RESTARTS AGE
redis-no-redis-cluster-0 1/1 Running 0 25m
redis-no-redis-cluster-1 1/1 Running 0 25m
redis-no-redis-cluster-2 1/1 Running 0 25m
redis-no-redis-cluster-3 1/1 Running 0 25m
redis-no-redis-cluster-4 1/1 Running 0 25m
redis-no-redis-cluster-5 1/1 Running 0 25m
Which chart: redis-cluster 6.3.9
Describe the bug Creating a redis cluster without password fails. The liveness and readiness probe fails for containers during creation.
To Reproduce helm upgrade redis bitnami/redis-cluster --install --namespace redis --create-namespace --set fullnameOverride=redis --set usePassword=false
Expected behavior Redis containers liveness and readiness probe does not fail during startup.
Version of Helm and Kubernetes:
helm version
:kubectl version
:Additional context Add any other context about the problem here.
From kubectl describe: