search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.84k stars 9.13k forks source link

[bitnami/redis-cluster] Deployment doesn't work (cluster in fail state) #12901

Closed baptiste-gaillet closed 1 year ago

baptiste-gaillet commented 1 year ago

Name and Version

bitnami/redis-cluster:8.2.4

What steps will reproduce the bug?

Run : 

$ helm repo add redis-cluster https://charts.bitnami.com/bitnami
$ helm install redis-cluster redis-cluster/redis-cluster --namespace=redis-cluster --create-namespace

Are you using any custom parameters or values?

Try with custom values and with the original helm values file.

What is the expected behavior?

Have a cluster up and running

What do you see instead?

Cluster is in fail mode, with an event :

Additional information

No response

baptiste-gaillet commented 1 year ago

Same probleme here, but not resolved and closed : https://github.com/bitnami/charts/issues/10172

fmulero commented 1 year ago

Thanks @baptiste-gaillet for opening this issue and share your findings.

Could share a kubectl describe of your failing pods? Also could you share more details from your kubernetes cluster? What storage class are you using by default?

baptiste-gaillet commented 1 year ago

You're welcome @fmulero ;p

I'm using the default "Retain" storageClass of my provider (Scaleway) : "scw-bssd-retain"

Here the: kubectl describe pod redis-cluster-0 but I have the 3 pods who are failing with the same error (redis-cluster-1 and redis-cluster-2)

$  kubectl describe pod redis-cluster-0

Name:         redis-cluster-0
Namespace:    redis-cluster
Priority:     0
Node:         scw-k8s-project-dev-pool-1-23-6-data-2695a2ec/10.194.79.83
Start Time:   Tue, 11 Oct 2022 19:20:41 +0200
Labels:       app.kubernetes.io/instance=redis-cluster
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=redis-cluster
              controller-revision-hash=redis-cluster-5bdd7446bd
              helm.sh/chart=redis-cluster-8.2.4
              statefulset.kubernetes.io/pod-name=redis-cluster-0
Annotations:  checksum/config: 791f2d511bc8ffaff547e08766c5725d110e41d23d360d0e731b068a875b1821
              checksum/scripts: 132e2e3eb40f8abcaa307d7719f11f2219728e6e369f41acaa7403bdbacb3b35
              checksum/secret: 49f85ea6ace2f4d18d44d5540cc9a71b9b8bde9e66222a449884d05ba4662c49
Status:       Running
IP:           100.64.5.238
IPs:
  IP:           100.64.5.238
Controlled By:  StatefulSet/redis-cluster
Containers:
  redis-cluster:
    Container ID:  containerd://b30d75089b09244137e45f54c2075634018e08154fc4914bea957f5a8dc7cf2a
    Image:         docker.io/bitnami/redis-cluster:7.0.5-debian-11-r6
    Image ID:      docker.io/bitnami/redis-cluster@sha256:6702448370278d0bb84b50ae16fea8e9c7843b9c4330c280055931c36f798045
    Ports:         6379/TCP, 16379/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /bin/bash
      -c
    Args:
      # Backwards compatibility change
      if ! [[ -f /opt/bitnami/redis/etc/redis.conf ]]; then
          echo COPYING FILE
          cp  /opt/bitnami/redis/etc/redis-default.conf /opt/bitnami/redis/etc/redis.conf
      fi
      pod_index=($(echo "$POD_NAME" | tr "-" "\n"))
      pod_index="${pod_index[-1]}"
      if [[ "$pod_index" == "0" ]]; then
        export REDIS_CLUSTER_CREATOR="yes"
        export REDIS_CLUSTER_REPLICAS="1"
      fi
      /opt/bitnami/scripts/redis-cluster/entrypoint.sh /opt/bitnami/scripts/redis-cluster/run.sh

    State:          Running
      Started:      Tue, 11 Oct 2022 19:20:49 +0200
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   128Mi
    Liveness:   exec [sh -c /scripts/ping_liveness_local.sh 5] delay=10s timeout=6s period=5s #success=1 #failure=5
    Readiness:  exec [sh -c /scripts/ping_readiness_local.sh 1] delay=10s timeout=2s period=5s #success=1 #failure=5
    Environment:
      POD_NAME:           redis-cluster-0 (v1:metadata.name)
      REDIS_NODES:        redis-cluster-0.redis-cluster-headless redis-cluster-1.redis-cluster-headless redis-cluster-2.redis-cluster-headless
      REDISCLI_AUTH:      <set to the key 'redis-password' in secret 'redis-cluster'>  Optional: false
      REDIS_PASSWORD:     <set to the key 'redis-password' in secret 'redis-cluster'>  Optional: false
      REDIS_AOF_ENABLED:  yes
      REDIS_TLS_ENABLED:  no
      REDIS_PORT:         6379
    Mounts:
      /bitnami/redis/data from redis-data (rw)
      /opt/bitnami/redis/etc/ from redis-tmp-conf (rw)
      /opt/bitnami/redis/etc/redis-default.conf from default-config (rw,path="redis-default.conf")
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-h2mbt (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  redis-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  redis-data-redis-cluster-0
    ReadOnly:   false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-cluster-scripts
    Optional:  false
  default-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-cluster-default
    Optional:  false
  redis-tmp-conf:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-h2mbt:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age               From                     Message
  ----     ------                  ----              ----                     -------
  Warning  FailedScheduling        52s               default-scheduler        0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
  Warning  FailedScheduling        50s               default-scheduler        0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
  Normal   Scheduled               48s               default-scheduler        Successfully assigned redis-cluster/redis-cluster-0 to scw-k8s-project-dev-pool-1-23-6-data-2695a2ec
  Normal   SuccessfulAttachVolume  46s               attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-41911e1e-cc40-4435-bcf4-bb98003dd64e"
  Normal   Pulled                  40s               kubelet                  Container image "docker.io/bitnami/redis-cluster:7.0.5-debian-11-r6" already present on machine
  Normal   Created                 40s               kubelet                  Created container redis-cluster
  Normal   Started                 40s               kubelet                  Started container redis-cluster
  Warning  Unhealthy               3s (x6 over 28s)  kubelet                  Readiness probe failed: cluster_state:fail

Thx again for your help. Baptiste.

fmulero commented 1 year ago

Thanks @baptiste-gaillet

We are going to need more information. Could you set image.debug=true and share also the logs from the pod?

baptiste-gaillet commented 1 year ago

Hi @fmulero .

Thx again for your help. Sure, here the logs of one of my pods (I have the same logs on the 3 pods) :

[94mredis-cluster-2 redis-cluster 1:C 13 Oct 2022 10:20:04.091 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-cluster-2 redis-cluster 1:C 13 Oct 2022 10:20:04.091 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
redis-cluster-2 redis-cluster 1:C 13 Oct 2022 10:20:04.091 # Configuration loaded
redis-cluster-2 redis-cluster 1:M 13 Oct 2022 10:20:04.091 * monotonic clock: POSIX clock_gettime
redis-cluster-2 redis-cluster 1:M 13 Oct 2022 10:20:04.092 * Node configuration loaded, I'm 35e52008b47f14062c434160414f0612a30ca29a
redis-cluster-2 redis-cluster                 _._
redis-cluster-2 redis-cluster            _.-``__ ''-._
redis-cluster-2 redis-cluster       _.-``    `.  `_.  ''-._           Redis 7.0.5 (00000000/0) 64 bit
redis-cluster-2 redis-cluster   .-`` .-```.  ```\/    _.,_ ''-._
redis-cluster-2 redis-cluster  (    '      ,       .-`  | `,    )     Running in cluster mode
redis-cluster-2 redis-cluster  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
redis-cluster-2 redis-cluster  |    `-._   `._    /     _.-'    |     PID: 1
redis-cluster-2 redis-cluster   `-._    `-._  `-./  _.-'    _.-'
redis-cluster-2 redis-cluster  |`-._`-._    `-.__.-'    _.-'_.-'|
redis-cluster-2 redis-cluster  |    `-._`-._        _.-'_.-'    |           https://redis.io
redis-cluster-2 redis-cluster   `-._    `-._`-.__.-'_.-'    _.-'
redis-cluster-2 redis-cluster  |`-._`-._    `-.__.-'    _.-'_.-'|
redis-cluster-2 redis-cluster  |    `-._`-._        _.-'_.-'    |
redis-cluster-2 redis-cluster   `-._    `-._`-.__.-'_.-'    _.-'
redis-cluster-2 redis-cluster       `-._    `-.__.-'    _.-'
redis-cluster-2 redis-cluster           `-._        _.-'
redis-cluster-2 redis-cluster               `-.__.-'
redis-cluster-2 redis-cluster
redis-cluster-2 redis-cluster 1:M 13 Oct 2022 10:20:04.093 # Server initialized
redis-cluster-2 redis-cluster 1:M 13 Oct 2022 10:20:04.096 * Reading RDB base file on AOF loading...
redis-cluster-2 redis-cluster 1:M 13 Oct 2022 10:20:04.096 * Loading RDB produced by version 7.0.5
redis-cluster-2 redis-cluster 1:M 13 Oct 2022 10:20:04.096 * RDB age 147551 seconds
redis-cluster-2 redis-cluster 1:M 13 Oct 2022 10:20:04.096 * RDB memory usage when created 1.51 Mb
redis-cluster-2 redis-cluster 1:M 13 Oct 2022 10:20:04.096 * RDB is base AOF
redis-cluster-2 redis-cluster 1:M 13 Oct 2022 10:20:04.096 * Done loading RDB, keys loaded: 0, keys expired: 0.
redis-cluster-2 redis-cluster 1:M 13 Oct 2022 10:20:04.096 * DB loaded from base file appendonly.aof.1.base.rdb: 0.001 seconds
redis-cluster-2 redis-cluster 1:M 13 Oct 2022 10:20:04.096 * DB loaded from append only file: 0.001 seconds
redis-cluster-2 redis-cluster 1:M 13 Oct 2022 10:20:04.096 * Opening AOF incr file appendonly.aof.1.incr.aof on server start
redis-cluster-2 redis-cluster 1:M 13 Oct 2022 10:20:04.097 * Ready to accept connections

For me it's just the probe who fail, so the cluster is never really up.

But the logs were the same, with or without the option image.debug=true, I think. Thx have a good day. Baptiste.

javsalgar commented 1 year ago

Hi,

Something tells me that it could be related to the PVC technology. Did you try it in a different cluster? This is to make sure that the issue is there.

baptiste-gaillet commented 1 year ago

Hi @javsalgar

Same fail (cluster_state:fail) with a gcp cluster and a gcp storage class ;(

Logs for one of pod : 

COPYING FILE                                                                                                                                                    │
│ redis-cluster 06:58:18.99                                                                                                                                       │
│ redis-cluster 06:58:18.99 Welcome to the Bitnami redis-cluster container                                                                                        │
│ redis-cluster 06:58:18.99 Subscribe to project updates by watching https://github.com/bitnami/containers                                                        │
│ redis-cluster 06:58:19.00 Submit issues and feature requests at https://github.com/bitnami/containers/issues                                                    │
│ redis-cluster 06:58:19.00                                                                                                                                       │
│ redis-cluster 06:58:19.00 INFO  ==> ** Starting Redis setup **                                                                                                  │
│ redis-cluster 06:58:19.03 INFO  ==> Initializing Redis                                                                                                          │
│ redis-cluster 06:58:19.04 INFO  ==> Setting Redis config file                                                                                                   │
│ Changing old IP 10.32.3.115 by the new one 10.32.3.116                                                                                                          │
│ Changing old IP 10.32.3.114 by the new one 10.32.2.108                                                                                                          │
│ Changing old IP 10.32.2.106 by the new one 10.32.3.117                                                                                                          │
│ redis-cluster 06:58:24.16 INFO  ==> ** Redis setup finished! **                                                                                                 │
│                                                                                                                                                                 │
│ 1:C 19 Oct 2022 06:58:24.196 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo                                                                                    │
│ 1:C 19 Oct 2022 06:58:24.196 # Redis version=7.0.5, bits=64, commit=00000000, modified=0, pid=1, just started                                                   │
│ 1:C 19 Oct 2022 06:58:24.196 # Configuration loaded                                                                                                             │
│ 1:M 19 Oct 2022 06:58:24.196 * monotonic clock: POSIX clock_gettime                                                                                             │
│ 1:M 19 Oct 2022 06:58:24.197 * Node configuration loaded, I'm 9db16e1ad11ed12137396499e535e6d66aad61d5                                                          │
│                 _._                                                                                                                                             │
│            _.-``__ ''-._                                                                                                                                        │
│       _.-``    `.  `_.  ''-._           Redis 7.0.5 (00000000/0) 64 bit                                                                                         │
│   .-`` .-```.  ```\/    _.,_ ''-._                                                                                                                              │
│  (    '      ,       .-`  | `,    )     Running in cluster mode                                                                                                 │
│  |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379                                                                                                              │
│  |    `-._   `._    /     _.-'    |     PID: 1                                                                                                        │
│   `-._    `-._  `-./  _.-'    _.-'                                                                                                                              │
│  |`-._`-._    `-.__.-'    _.-'_.-'|                                                                                                                             │
│  |    `-._`-._        _.-'_.-'    |           https://redis.io/                                                                                                  │
│   `-._    `-._`-.__.-'_.-'    _.-'                                                                                                                              │
│  |`-._`-._    `-.__.-'    _.-'_.-'|                                                                                                                             │
│  |    `-._`-._        _.-'_.-'    |                                                                                                                             │
│   `-._    `-._`-.__.-'_.-'    _.-'                                                                                                                              │
│       `-._    `-.__.-'    _.-'                                                                                                                                  │
│           `-._        _.-'                                                                                                                                      │
│               `-.__.-'                                                                                                                                          │
│                                                                                                                                                                 │
│ 1:M 19 Oct 2022 06:58:24.198 # Server initialized                                                                                                               │
│ 1:M 19 Oct 2022 06:58:24.200 * Reading RDB base file on AOF loading...                                                                                          │
│ 1:M 19 Oct 2022 06:58:24.200 * Loading RDB produced by version 7.0.5                                                                                            │
│ 1:M 19 Oct 2022 06:58:24.200 * RDB age 513 seconds                                                                                                              │
│ 1:M 19 Oct 2022 06:58:24.200 * RDB memory usage when created 1.51 Mb                                                                                            │
│ 1:M 19 Oct 2022 06:58:24.200 * RDB is base AOF                                                                                                                  │
│ 1:M 19 Oct 2022 06:58:24.200 * Done loading RDB, keys loaded: 0, keys expired: 0.                                                                               │
│ 1:M 19 Oct 2022 06:58:24.200 * DB loaded from base file appendonly.aof.1.base.rdb: 0.001 seconds                                                                │
│ 1:M 19 Oct 2022 06:58:24.200 * DB loaded from append only file: 0.001 seconds                                                                                   │
│ 1:M 19 Oct 2022 06:58:24.200 * Opening AOF incr file appendonly.aof.1.incr.aof on server start                                                                  │
│ 1:M 19 Oct 2022 06:58:24.200 * Ready to accept connections

Describe for one pod :

Name:         redis-cluster-2
Namespace:    redis-cluster
Priority:     0
Node:         gke-k8s-dev-pool-1-21-6-gke1500-a58c9898-xpl9/10.132.0.39
Start Time:   Wed, 19 Oct 2022 08:58:09 +0200
Labels:       app.kubernetes.io/instance=redis-cluster
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=redis-cluster
              controller-revision-hash=redis-cluster-6f756f9d7
              helm.sh/chart=redis-cluster-8.2.4
              statefulset.kubernetes.io/pod-name=redis-cluster-2
Annotations:  checksum/config: 791f2d511bc8ffaff547e08766c5725d110e41d23d360d0e731b068a875b1821
              checksum/scripts: 132e2e3eb40f8abcaa307d7719f11f2219728e6e369f41acaa7403bdbacb3b35
              checksum/secret: dc00b6624ec7ad401d6803f61e0f9b66f2c1f0a7d483d6b18475a532b5c6408b
Status:       Running
IP:           10.32.3.117
IPs:
  IP:           10.32.3.117
Controlled By:  StatefulSet/redis-cluster
Containers:
  redis-cluster:
    Container ID:  containerd://9d370882c598e84ef2409bc15d1745e6f297528ede9a3eef4cf4c4d56f7d2d94
    Image:         docker.io/bitnami/redis-cluster:7.0.5-debian-11-r6
    Image ID:      docker.io/bitnami/redis-cluster@sha256:6702448370278d0bb84b50ae16fea8e9c7843b9c4330c280055931c36f798045
    Ports:         6379/TCP, 16379/TCP
    Host Ports:    0/TCP, 0/TCP
    Command:
      /bin/bash
      -c
    Args:
      # Backwards compatibility change
      if ! [[ -f /opt/bitnami/redis/etc/redis.conf ]]; then
          echo COPYING FILE
          cp  /opt/bitnami/redis/etc/redis-default.conf /opt/bitnami/redis/etc/redis.conf
      fi
      pod_index=($(echo "$POD_NAME" | tr "-" "\n"))
      pod_index="${pod_index[-1[]}"
      if [[ "$pod_index" == "0" ]]; then
        export REDIS_CLUSTER_CREATOR="yes"
        export REDIS_CLUSTER_REPLICAS="1"
      fi
      /opt/bitnami/scripts/redis-cluster/entrypoint.sh /opt/bitnami/scripts/redis-cluster/run.sh

    State:          Running
      Started:      Wed, 19 Oct 2022 08:58:18 +0200
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
      memory:   128Mi
    Liveness:   exec [sh -c /scripts/ping_liveness_local.sh 5] delay=10s timeout=6s period=5s #success=1 #failure=5
    Readiness:  exec [sh -c /scripts/ping_readiness_local.sh 1] delay=10s timeout=2s period=5s #success=1 #failure=5
    Environment:
      POD_NAME:           redis-cluster-2 (v1:metadata.name)
      REDIS_NODES:        redis-cluster-0.redis-cluster-headless redis-cluster-1.redis-cluster-headless redis-cluster-2.redis-cluster-headless 
      REDISCLI_AUTH:      <set to the key 'redis-password' in secret 'redis-cluster'>  Optional: false
      REDIS_PASSWORD:     <set to the key 'redis-password' in secret 'redis-cluster'>  Optional: false
      REDIS_AOF_ENABLED:  yes
      REDIS_TLS_ENABLED:  no
      REDIS_PORT:         6379
    Mounts:
      /bitnami/redis/data from redis-data (rw)
      /opt/bitnami/redis/etc/ from redis-tmp-conf (rw)
      /opt/bitnami/redis/etc/redis-default.conf from default-config (rw,path="redis-default.conf")
      /scripts from scripts (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-522qc (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  redis-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  redis-data-redis-cluster-2
    ReadOnly:   false
  scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-cluster-scripts
    Optional:  false
  default-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      redis-cluster-default
    Optional:  false
  redis-tmp-conf:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-522qc:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                   From                     Message
  ----     ------                  ----                  ----                     -------
  Normal   Scheduled               16m                   default-scheduler        Successfully assigned redis-cluster/redis-cluster-2 to gke-k8s-dev-pool-1-21-6-gke1500-a58c9898-xpl9
  Normal   SuccessfulAttachVolume  16m                   attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-e1a67bd6-93a1-47cd-b0d0-a3b76e1d63cb"
  Normal   Pulled                  16m                   kubelet                  Container image "docker.io/bitnami/redis-cluster:7.0.5-debian-11-r6" already present on machine
  Normal   Created                 16m                   kubelet                  Created container redis-cluster
  Normal   Started                 16m                   kubelet                  Started container redis-cluster
  Warning  Unhealthy               106s (x179 over 16m)  kubelet                  Readiness probe failed: cluster_state:fail
baptiste-gaillet commented 1 year ago

My values.yaml file if needed :

## @section Global parameters
## Global Docker image parameters
## Please, note that this will override the image parameters, including dependencies, configured to use the global value
## Current available global Docker image parameters: imageRegistry, imagePullSecrets and storageClass
##

## @param global.imageRegistry Global Docker image registry
## @param global.imagePullSecrets Global Docker registry secret names as an array
## @param global.storageClass Global StorageClass for Persistent Volume(s)
## @param global.redis.password Redis&reg; password (overrides `password`)
##
global:
  imageRegistry: ""
  ## E.g.
  ## imagePullSecrets:
  ##   - myRegistryKeySecretName
  ##
  imagePullSecrets: []
  storageClass: "slow-eu-west1-bc-retain" # "scw-bssd-retain"
  redis:
    password: ""

## @section Redis&reg; Cluster Common parameters
##

## @param nameOverride String to partially override common.names.fullname template (will maintain the release name)
##
nameOverride: ""
## @param fullnameOverride String to fully override common.names.fullname template
##
fullnameOverride: ""
## @param clusterDomain Kubernetes Cluster Domain
##
clusterDomain: cluster.local
## @param commonAnnotations Annotations to add to all deployed objects
##
commonAnnotations: {}
## @param commonLabels Labels to add to all deployed objects
##
commonLabels: {}
## @param extraDeploy Array of extra objects to deploy with the release (evaluated as a template)
##
extraDeploy: []

## Enable diagnostic mode in the deployment
##
diagnosticMode:
  ## @param diagnosticMode.enabled Enable diagnostic mode (all probes will be disabled and the command will be overridden)
  ##
  enabled: false
  ## @param diagnosticMode.command Command to override all containers in the deployment
  ##
  command:
    - sleep
  ## @param diagnosticMode.args Args to override all containers in the deployment
  ##
  args:
    - infinity

## Bitnami Redis&reg; image version
## ref: https://hub.docker.com/r/bitnami/redis/tags/
## @param image.registry Redis&reg; cluster image registry
## @param image.repository Redis&reg; cluster image repository
## @param image.tag Redis&reg; cluster image tag (immutable tags are recommended)
## @param image.digest Redis&reg; cluster image digest in the way sha256:aa.... Please note this parameter, if set, will override the tag
## @param image.pullPolicy Redis&reg; cluster image pull policy
## @param image.pullSecrets Specify docker-registry secret names as an array
## @param image.debug Enable image debug mode
##
image:
  registry: docker.io
  repository: bitnami/redis-cluster
  ## Bitnami Redis&reg; image tag
  ## ref: https://github.com/bitnami/containers/tree/main/bitnami/redis#supported-tags-and-respective-dockerfile-links
  ##
  tag: 7.0.5-debian-11-r6
  digest: ""
  ## Specify a imagePullPolicy
  ## Defaults to 'Always' if image tag is 'latest', else set to 'IfNotPresent'
  ## ref: https://kubernetes.io/docs/user-guide/images/#pre-pulling-images
  ##
  pullPolicy: IfNotPresent
  ## Optionally specify an array of imagePullSecrets.
  ## Secrets must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ## e.g:
  ## pullSecrets:
  ##   - myRegistryKeySecretName
  ##
  pullSecrets: []
  ## Enable debug mode
  ##
  debug: false
## Network Policy
## @param networkPolicy.enabled Enable NetworkPolicy
## @param networkPolicy.allowExternal The Policy model to apply. Don't require client label for connections
## @param networkPolicy.ingressNSMatchLabels Allow connections from other namespacess. Just set label for namespace and set label for pods (optional).
## @param networkPolicy.ingressNSPodMatchLabels For other namespaces match by pod labels and namespace labels
##
networkPolicy:
  enabled: false
  ## When set to false, only pods with the correct
  ## client label will have network access to the port Redis&reg; is listening
  ## on. When true, Redis&reg; will accept connections from any source
  ## (with the correct destination port).
  ##
  allowExternal: true
  ingressNSMatchLabels: {}
  ingressNSPodMatchLabels: {}

serviceAccount:
  ## @param serviceAccount.create Specifies whether a ServiceAccount should be created
  ##
  create: false
  ## @param serviceAccount.name The name of the ServiceAccount to create
  ## If not set and create is true, a name is generated using the fullname template
  ##
  name: ""
  ## @param serviceAccount.annotations Annotations for Cassandra Service Account
  ##
  annotations: {}
  ## @param serviceAccount.automountServiceAccountToken Automount API credentials for a service account.
  ##
  automountServiceAccountToken: false

rbac:
  ## @param rbac.create Specifies whether RBAC resources should be created
  ##
  create: false
  role:
    ## @param rbac.role.rules Rules to create. It follows the role specification
    ## rules:
    ##  - apiGroups:
    ##    - extensions
    ##    resources:
    ##      - podsecuritypolicies
    ##    verbs:
    ##      - use
    ##    resourceNames:
    ##      - gce.unprivileged
    ##
    rules: []
## Redis&reg; pod Security Context
## @param podSecurityContext.enabled Enable Redis&reg; pod Security Context
## @param podSecurityContext.fsGroup Group ID for the pods
## @param podSecurityContext.runAsUser User ID for the pods
## @param podSecurityContext.sysctls Set namespaced sysctls for the pods
##
podSecurityContext:
  enabled: true
  fsGroup: 1001
  runAsUser: 1001
  ## Uncomment the setting below to increase the net.core.somaxconn value
  ## e.g:
  ## sysctls:
  ##   - name: net.core.somaxconn
  ##     value: "10000"
  ##
  sysctls: []
## @param podDisruptionBudget Limits the number of pods of the replicated application that are down simultaneously from voluntary disruptions
## ref: https://kubernetes.io/docs/concepts/workloads/pods/disruptions
##
podDisruptionBudget: {}
## @param minAvailable Min number of pods that must still be available after the eviction
##
minAvailable: ""
## @param maxUnavailable Max number of pods that can be unavailable after the eviction
##
maxUnavailable: ""
## Containers Security Context
## @param containerSecurityContext.enabled Enable Containers' Security Context
## @param containerSecurityContext.runAsUser User ID for the containers.
## @param containerSecurityContext.runAsNonRoot Run container as non root
##
containerSecurityContext:
  enabled: true
  runAsUser: 1001
  runAsNonRoot: true
## @param usePassword Use password authentication
##
usePassword: true
## @param password Redis&reg; password (ignored if existingSecret set)
## Defaults to a random 10-character alphanumeric string if not set and usePassword is true
## ref: https://github.com/bitnami/containers/tree/main/bitnami/redis#setting-the-server-password-on-first-run
##
password: ""
## @param existingSecret Name of existing secret object (for password authentication)
##
existingSecret: ""
# existingSecret: "redis-cluster-secrets"
## @param existingSecretPasswordKey Name of key containing password to be retrieved from the existing secret
##
existingSecretPasswordKey: ""
# existingSecretPasswordKey: "REDIS_PASSWORD"
## @param usePasswordFile Mount passwords as files instead of environment variables
##
usePasswordFile: false
##
## TLS configuration
##
tls:
  ## @param tls.enabled Enable TLS support for replication traffic
  ##
  enabled: false
  ## @param tls.authClients Require clients to authenticate or not
  ##
  authClients: true
  ## @param tls.autoGenerated Generate automatically self-signed TLS certificates
  ##
  autoGenerated: false
  ## @param tls.existingSecret The name of the existing secret that contains the TLS certificates
  ##
  existingSecret: ""
  ## @param tls.certificatesSecret DEPRECATED. Use tls.existingSecret instead
  ##
  certificatesSecret: ""
  ## @param tls.certFilename Certificate filename
  ##
  certFilename: ""
  ## @param tls.certKeyFilename Certificate key filename
  ##
  certKeyFilename: ""
  ## @param tls.certCAFilename CA Certificate filename
  ##
  certCAFilename: ""
  ## @param tls.dhParamsFilename File containing DH params (in order to support DH based ciphers)
  ##
  dhParamsFilename: ""
## Redis&reg; Service properties for standalone mode.
##
service:
  ## @param service.ports.redis Kubernetes Redis service port
  ##
  ports:
    redis: 6379
  ## Node ports to expose
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport
  ## @param service.nodePorts.redis Node port for Redis
  ##
  nodePorts:
    redis: ""
  ## @param service.extraPorts Extra ports to expose in the service (normally used with the `sidecar` value)
  ##
  extraPorts: []
  ## @param service.annotations Provide any additional annotations which may be required.
  ## This can be used to set the LoadBalancer service type to internal only.
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#internal-load-balancer
  ##
  annotations: {}
  ## @param service.labels Additional labels for redis service
  ##
  labels: {}
  ## @param service.type Service type for default redis service
  ## Setting this to LoadBalancer may require corresponding service annotations for loadbalancer creation to succeed.
  ## Currently supported types are ClusterIP (default) and LoadBalancer
  ##
  type: ClusterIP
  ## @param service.clusterIP Service Cluster IP
  ## e.g.:
  ## clusterIP: None
  ##
  clusterIP: ""
  ## @param service.loadBalancerIP Load balancer IP if `service.type` is `LoadBalancer`
  ## If service.type is LoadBalancer, request a specific static IP address if supported by the cloud provider, otherwise leave blank
  ##
  loadBalancerIP: ""
  ## @param service.loadBalancerSourceRanges Service Load Balancer sources
  ## ref: https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service
  ## e.g:
  ## loadBalancerSourceRanges:
  ##   - 10.10.10.0/24
  ##
  loadBalancerSourceRanges: []
  ## @param service.externalTrafficPolicy Service external traffic policy
  ## ref https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip
  ##
  externalTrafficPolicy: Cluster
  ## @param service.sessionAffinity Session Affinity for Kubernetes service, can be "None" or "ClientIP"
  ## If "ClientIP", consecutive client requests will be directed to the same Pod
  ## ref: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
  ##
  sessionAffinity: None
  ## @param service.sessionAffinityConfig Additional settings for the sessionAffinity
  ## sessionAffinityConfig:
  ##   clientIP:
  ##     timeoutSeconds: 300
  ##
  sessionAffinityConfig: {}
## Enable persistence using Persistent Volume Claims
## ref: https://kubernetes.io/docs/user-guide/persistent-volumes/
##
persistence:
  ## @param persistence.path Path to mount the volume at, to use other images Redis&reg; images.
  ##
  path: /bitnami/redis/data
  ## @param persistence.subPath The subdirectory of the volume to mount to, useful in dev environments and one PV for multiple services
  ##
  subPath: ""
  ## @param persistence.storageClass Storage class of backing PVC
  ## If defined, storageClassName: <storageClass>
  ## If set to "-", storageClassName: "", which disables dynamic provisioning
  ## If undefined (the default) or set to null, no storageClassName spec is
  ##   set, choosing the default provisioner.  (gp2 on AWS, standard on
  ##   GKE, AWS & OpenStack)
  ##
  storageClass: "scw-bssd-retain"
  ## @param persistence.annotations Persistent Volume Claim annotations
  ##
  annotations: {}
  ## @param persistence.accessModes Persistent Volume Access Modes
  ##
  accessModes:
    - ReadWriteOnce
  ## @param persistence.size Size of data volume
  ##
  size: 6Gi
  ## @param persistence.matchLabels Persistent Volume selectors
  ## https://kubernetes.io/docs/concepts/storage/persistent-volumes/#selector
  ##
  matchLabels: {}
  ## @param persistence.matchExpressions matchExpressions Persistent Volume selectors
  ##
  matchExpressions: {}

## Init containers parameters:
## volumePermissions: Change the owner of the persist volume mountpoint to RunAsUser:fsGroup
##
volumePermissions:
  ## @param volumePermissions.enabled Enable init container that changes volume permissions in the registry (for cases where the default k8s `runAsUser` and `fsUser` values do not work)
  ##
  enabled: false
  ## @param volumePermissions.image.registry Init container volume-permissions image registry
  ## @param volumePermissions.image.repository Init container volume-permissions image repository
  ## @param volumePermissions.image.tag Init container volume-permissions image tag
  ## @param volumePermissions.image.digest Init container volume-permissions image digest in the way sha256:aa.... Please note this parameter, if set, will override the tag
  ## @param volumePermissions.image.pullPolicy Init container volume-permissions image pull policy
  ## @param volumePermissions.image.pullSecrets Specify docker-registry secret names as an array
  ##
  image:
    registry: docker.io
    repository: bitnami/bitnami-shell
    tag: 11-debian-11-r40
    digest: ""
    pullPolicy: IfNotPresent
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ## e.g:
    ## pullSecrets:
    ##   - myRegistryKeySecretName
    ##
    pullSecrets: []
  ## Container resource requests and limits
  ## ref: https://kubernetes.io/docs/user-guide/compute-resources/
  ## @param volumePermissions.resources.limits The resources limits for the container
  ## @param volumePermissions.resources.requests The requested resources for the container
  ##
  resources:
    ## Example:
    ## limits:
    ##    cpu: 100m
    ##    memory: 128Mi
    ##
    limits: {}
    ## Examples:
    ## requests:
    ##    cpu: 100m
    ##    memory: 128Mi
    ##
    requests: {}
## PodSecurityPolicy configuration
## ref: https://kubernetes.io/docs/concepts/policy/pod-security-policy/
## @param podSecurityPolicy.create Whether to create a PodSecurityPolicy. WARNING: PodSecurityPolicy is deprecated in Kubernetes v1.21 or later, unavailable in v1.25 or later
##
podSecurityPolicy:
  create: false

## @section Redis&reg; statefulset parameters
##

redis:
  ## @param redis.command Redis&reg; entrypoint string. The command `redis-server` is executed if this is not provided
  ##
  command: []
  ## @param redis.args Arguments for the provided command if needed
  ##
  args: []
  ## @param redis.updateStrategy.type Argo Workflows statefulset strategy type
  ## ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies
  ##
  updateStrategy:
    ## StrategyType
    ## Can be set to RollingUpdate or OnDelete
    ##
    type: RollingUpdate
    ## @param redis.updateStrategy.rollingUpdate.partition Partition update strategy
    ## https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#partitions
    ##
    rollingUpdate:
      partition: 0

  ## @param redis.podManagementPolicy Statefulset Pod management policy, it needs to be Parallel to be able to complete the cluster join
  ## Ref: https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#pod-management-policies
  ##
  podManagementPolicy: Parallel
  ## @param redis.hostAliases Deployment pod host aliases
  ## https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/
  ##
  hostAliases: []
  ## @param redis.hostNetwork Host networking requested for this pod. Use the host's network namespace.
  ## https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.22/#podspec-v1-core
  ##
  hostNetwork: false
  ## @param redis.useAOFPersistence Whether to use AOF Persistence mode or not
  ## It is strongly recommended to use this type when dealing with clusters
  ## ref: https://redis.io/topics/persistence#append-only-file
  ## ref: https://redis.io/topics/cluster-tutorial#creating-and-using-a-redis-cluster
  ##
  useAOFPersistence: "yes"
  ## @param redis.containerPorts.redis Redis&reg; port
  ## @param redis.containerPorts.bus The busPort should be obtained adding 10000 to the redisPort. By default: 10000 + 6379 = 16379
  ##
  containerPorts:
    redis: 6379
    bus: 16379
  ## @param redis.lifecycleHooks LifecycleHook to set additional configuration before or after startup. Evaluated as a template
  ##
  lifecycleHooks: {}
  ## @param redis.extraVolumes Extra volumes to add to the deployment
  ##
  extraVolumes: []
  ## @param redis.extraVolumeMounts Extra volume mounts to add to the container
  ##
  extraVolumeMounts: []
  ## @param redis.customLivenessProbe Override default liveness probe
  ##
  customLivenessProbe: {}
  ## @param redis.customReadinessProbe Override default readiness probe
  ##
  customReadinessProbe: {}
  ## @param redis.customStartupProbe Custom startupProbe that overrides the default one
  ##
  customStartupProbe: {}
  ## @param redis.initContainers Extra init containers to add to the deployment
  ##
  initContainers: []
  ## @param redis.sidecars Extra sidecar containers to add to the deployment
  ##
  sidecars: []
  ## @param redis.podLabels Additional labels for Redis&reg; pod
  ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
  ##
  podLabels: {}
  ## @param redis.priorityClassName Redis&reg; Master pod priorityClassName
  ##
  priorityClassName: ""
  ## @param redis.configmap Additional Redis&reg; configuration for the nodes
  ## ref: https://redis.io/topics/config
  ##
  configmap: ""
  ## @param redis.extraEnvVars An array to add extra environment variables
  ## For example:
  ##  - name: BEARER_AUTH
  ##    value: true
  ##
  extraEnvVars: []
  ## @param redis.extraEnvVarsCM ConfigMap with extra environment variables
  ##
  extraEnvVarsCM: ""
  ## @param redis.extraEnvVarsSecret Secret with extra environment variables
  ##
  extraEnvVarsSecret: ""
  ## @param redis.podAnnotations Redis&reg; additional annotations
  ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
  ##
  podAnnotations: {}
  ## Redis&reg; resource requests and limits
  ## ref: https://kubernetes.io/docs/user-guide/compute-resources/
  ## @param redis.resources.limits The resources limits for the container
  ## @param redis.resources.requests The requested resources for the container
  ##
  resources:
    ## Example:
    ## limits:
    ##    cpu: 100m
    ##    memory: 128Mi
    ##
    limits: {}
    ## Examples:
    ## requests:
    ##    cpu: 100m
    ##    memory: 128Mi
    ##
    requests:
      cpu: 100m
      memory: 128Mi
  ## @param redis.schedulerName Use an alternate scheduler, e.g. "stork".
  ## ref: https://kubernetes.io/docs/tasks/administer-cluster/configure-multiple-schedulers/
  ##
  schedulerName: ""
  ## @param redis.shareProcessNamespace Enable shared process namespace in a pod.
  ## If set to false (default), each container will run in separate namespace, redis will have PID=1.
  ## If set to true, the /pause will run as init process and will reap any zombie PIDs,
  ## for example, generated by a custom exec probe running longer than a probe timeoutSeconds.
  ## Enable this only if customLivenessProbe or customReadinessProbe is used and zombie PIDs are accumulating.
  ## Ref: https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/
  ##
  shareProcessNamespace: false
  ## Configure extra options for Redis&reg; liveness probes
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes)
  ## @param redis.livenessProbe.enabled Enable livenessProbe
  ## @param redis.livenessProbe.initialDelaySeconds Initial delay seconds for livenessProbe
  ## @param redis.livenessProbe.periodSeconds Period seconds for livenessProbe
  ## @param redis.livenessProbe.timeoutSeconds Timeout seconds for livenessProbe
  ## @param redis.livenessProbe.failureThreshold Failure threshold for livenessProbe
  ## @param redis.livenessProbe.successThreshold Success threshold for livenessProbe
  ##
  livenessProbe:
    enabled: true
    initialDelaySeconds: 10
    periodSeconds: 5
    timeoutSeconds: 5
    successThreshold: 1
    failureThreshold: 5
  ## Configure extra options for Redis&reg; readiness probes
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#configure-probes)
  ## @param redis.readinessProbe.enabled Enable readinessProbe
  ## @param redis.readinessProbe.initialDelaySeconds Initial delay seconds for readinessProbe
  ## @param redis.readinessProbe.periodSeconds Period seconds for readinessProbe
  ## @param redis.readinessProbe.timeoutSeconds Timeout seconds for readinessProbe
  ## @param redis.readinessProbe.failureThreshold Failure threshold for readinessProbe
  ## @param redis.readinessProbe.successThreshold Success threshold for readinessProbe
  ##
  readinessProbe:
    enabled: true
    initialDelaySeconds: 10
    periodSeconds: 5
    timeoutSeconds: 1
    successThreshold: 1
    failureThreshold: 5
  ## @param redis.startupProbe.enabled Enable startupProbe
  ## @param redis.startupProbe.path Path to check for startupProbe
  ## @param redis.startupProbe.initialDelaySeconds Initial delay seconds for startupProbe
  ## @param redis.startupProbe.periodSeconds Period seconds for startupProbe
  ## @param redis.startupProbe.timeoutSeconds Timeout seconds for startupProbe
  ## @param redis.startupProbe.failureThreshold Failure threshold for startupProbe
  ## @param redis.startupProbe.successThreshold Success threshold for startupProbe
  ##
  startupProbe:
    enabled: false
    path: /
    initialDelaySeconds: 300
    periodSeconds: 10
    timeoutSeconds: 5
    failureThreshold: 6
    successThreshold: 1
  ## @param redis.podAffinityPreset Redis&reg; pod affinity preset. Ignored if `redis.affinity` is set. Allowed values: `soft` or `hard`
  ## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity
  ##
  podAffinityPreset: ""
  ## @param redis.podAntiAffinityPreset Redis&reg; pod anti-affinity preset. Ignored if `redis.affinity` is set. Allowed values: `soft` or `hard`
  ## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity
  ##
  podAntiAffinityPreset: soft
  ## Redis&reg; node affinity preset
  ## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity
  ##
  nodeAffinityPreset:
    ## @param redis.nodeAffinityPreset.type Redis&reg; node affinity preset type. Ignored if `redis.affinity` is set. Allowed values: `soft` or `hard`
    ##
    type: ""
    ## @param redis.nodeAffinityPreset.key Redis&reg; node label key to match Ignored if `redis.affinity` is set.
    ## E.g.
    ## key: "kubernetes.io/e2e-az-name"
    ##
    key: ""
    ## @param redis.nodeAffinityPreset.values Redis&reg; node label values to match. Ignored if `redis.affinity` is set.
    ## E.g.
    ## values:
    ##   - e2e-az1
    ##   - e2e-az2
    ##
    values: []
  ## @param redis.affinity Affinity settings for Redis&reg; pod assignment
  ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
  ## Note: redis.podAffinityPreset, redis.podAntiAffinityPreset, and redis.nodeAffinityPreset will be ignored when it's set
  ##
  affinity: {}
  ## @param redis.nodeSelector Node labels for Redis&reg; pods assignment
  ## ref: https://kubernetes.io/docs/user-guide/node-selection/
  ##
  nodeSelector: {}
  ## @param redis.tolerations Tolerations for Redis&reg; pods assignment
  ## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
  ##
  tolerations: []
  ## @param redis.topologySpreadConstraints Pod topology spread constraints for Redis&reg; pod
  ## https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/
  ## The value is evaluated as a template
  ##
  topologySpreadConstraints: []

## @section Cluster update job parameters
##

## Cluster update job settings
##
updateJob:
  ## @param updateJob.activeDeadlineSeconds Number of seconds the Job to create the cluster will be waiting for the Nodes to be ready.
  ##
  activeDeadlineSeconds: 600
  ## @param updateJob.command Container command (using container default if not set)
  ##
  command: []
  ## @param updateJob.args Container args (using container default if not set)
  ##
  args: []
  ## @param updateJob.hostAliases Deployment pod host aliases
  ## https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/
  ##
  hostAliases: []
  ## @param updateJob.annotations Job annotations
  ##
  annotations: {}
  ## @param updateJob.podAnnotations Job pod annotations
  ##
  podAnnotations: {}
  ## @param updateJob.podLabels Pod extra labels
  ## ref: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/
  ##
  podLabels: {}
  ## @param updateJob.extraEnvVars An array to add extra environment variables
  ## For example:
  ##  - name: BEARER_AUTH
  ##    value: true
  ##
  extraEnvVars: []
  ## @param updateJob.extraEnvVarsCM ConfigMap containing extra environment variables
  ##
  extraEnvVarsCM: ""
  ## @param updateJob.extraEnvVarsSecret Secret containing extra environment variables
  ##
  extraEnvVarsSecret: ""
  ## @param updateJob.extraVolumes Extra volumes to add to the deployment
  ##
  extraVolumes: []
  ## @param updateJob.extraVolumeMounts Extra volume mounts to add to the container
  ##
  extraVolumeMounts: []
  ## @param updateJob.initContainers Extra init containers to add to the deployment
  ##
  initContainers: []
  ## @param updateJob.podAffinityPreset Update job pod affinity preset. Ignored if `updateJob.affinity` is set. Allowed values: `soft` or `hard`
  ## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity
  ##
  podAffinityPreset: ""
  ## @param updateJob.podAntiAffinityPreset Update job pod anti-affinity preset. Ignored if `updateJob.affinity` is set. Allowed values: `soft` or `hard`
  ## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#inter-pod-affinity-and-anti-affinity
  ##
  podAntiAffinityPreset: soft
  ## Update job node affinity preset
  ## ref: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/#node-affinity
  ##
  nodeAffinityPreset:
    ## @param updateJob.nodeAffinityPreset.type Update job node affinity preset type. Ignored if `updateJob.affinity` is set. Allowed values: `soft` or `hard`
    ##
    type: ""
    ## @param updateJob.nodeAffinityPreset.key Update job node label key to match Ignored if `updateJob.affinity` is set.
    ## E.g.
    ## key: "kubernetes.io/e2e-az-name"
    ##
    key: ""
    ## @param updateJob.nodeAffinityPreset.values Update job node label values to match. Ignored if `updateJob.affinity` is set.
    ## E.g.
    ## values:
    ##   - e2e-az1
    ##   - e2e-az2
    ##
    values: []
  ## @param updateJob.affinity Affinity for update job pods assignment
  ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
  ## Note: updateJob.podAffinityPreset, updateJob.podAntiAffinityPreset, and updateJob.nodeAffinityPreset will be ignored when it's set
  ##
  affinity: {}
  ## @param updateJob.nodeSelector Node labels for update job pods assignment
  ## ref: https://kubernetes.io/docs/user-guide/node-selection/
  ##
  nodeSelector: {}
  ## @param updateJob.tolerations Tolerations for update job pods assignment
  ## ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
  ##
  tolerations: []
  ## @param updateJob.priorityClassName Priority class name
  ##
  priorityClassName: ""
  ## Container resource requests and limits
  ## ref: https://kubernetes.io/docs/user-guide/compute-resources/
  ## We usually recommend not to specify default resources and to leave this as a conscious
  ## choice for the user. This also increases chances charts run on environments with little
  ## resources, such as Minikube. If you do want to specify resources, uncomment the following
  ## lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  ## @param updateJob.resources.limits The resources limits for the container
  ## @param updateJob.resources.requests The requested resources for the container
  ##
  resources:
    ## Example:
    ## limits:
    ##    cpu: 500m
    ##    memory: 1Gi
    ##
    limits: {}
    ## Examples:
    ## requests:
    ##    cpu: 250m
    ##    memory: 256Mi
    ##
    requests: {}

## @section Cluster management parameters
##

## Redis&reg; Cluster settings
##
cluster:
  ## @param cluster.init Enable the initialization of the Redis&reg; Cluster
  ##
  init: true
  ## Number of Redis&reg; nodes to be deployed
  ##
  ## Note:
  ## This is total number of nodes including the replicas. Meaning there will be 3 master and 3 replica
  ## nodes (as replica count is set to 1 by default, there will be 1 replica per master node).
  ## Hence, nodes = numberOfMasterNodes + numberOfMasterNodes * replicas
  ##
  ## @param cluster.nodes The number of master nodes should always be >= 3, otherwise cluster creation will fail
  ##
  nodes: 3
  ## @param cluster.replicas Number of replicas for every master in the cluster
  ## Parameter to be passed as --cluster-replicas to the redis-cli --cluster create
  ## 1 means that we want a replica for every master created
  ##
  replicas: 1
  ## Configuration to access the Redis&reg; Cluster from outside the Kubernetes cluster
  ##
  externalAccess:
    ## @param cluster.externalAccess.enabled Enable access to the Redis
    ##
    enabled: false
    service:
      ## @param cluster.externalAccess.service.type Type for the services used to expose every Pod
      ## At this moment only LoadBalancer is supported
      ##
      type: LoadBalancer
      ## @param cluster.externalAccess.service.port Port for the services used to expose every Pod
      ##
      port: 6379
      ## @param cluster.externalAccess.service.loadBalancerIP Array of load balancer IPs for each Redis&reg; node. Length must be the same as cluster.nodes
      ##
      loadBalancerIP: []
      ## @param cluster.externalAccess.service.loadBalancerSourceRanges Service Load Balancer sources
      ## ref: https://kubernetes.io/docs/tasks/access-application-cluster/configure-cloud-provider-firewall/#restrict-access-for-loadbalancer-service
      ## e.g:
      ## loadBalancerSourceRanges:
      ##   - 10.10.10.0/24
      ##
      loadBalancerSourceRanges: []
      ## @param cluster.externalAccess.service.annotations Annotations to add to the services used to expose every Pod of the Redis&reg; Cluster
      ##
      annotations: {}
  ## This section allows to update the Redis&reg; cluster nodes.
  ##
  update:
    ## @param cluster.update.addNodes Boolean to specify if you want to add nodes after the upgrade
    ## Setting this to true a hook will add nodes to the Redis&reg; cluster after the upgrade. currentNumberOfNodes and currentNumberOfReplicas is required
    ##
    addNodes: false
    ## @param cluster.update.currentNumberOfNodes Number of currently deployed Redis&reg; nodes
    ##
    currentNumberOfNodes: 6
    ## @param cluster.update.currentNumberOfReplicas Number of currently deployed Redis&reg; replicas
    ##
    currentNumberOfReplicas: 1
    ## @param cluster.update.newExternalIPs External IPs obtained from the services for the new nodes to add to the cluster
    ##
    newExternalIPs: []

## @section Metrics sidecar parameters
##

## Prometheus Exporter / Metrics
##
metrics:
  ## @param metrics.enabled Start a side-car prometheus exporter
  ##
  enabled: false
  ## @param metrics.image.registry Redis&reg; exporter image registry
  ## @param metrics.image.repository Redis&reg; exporter image name
  ## @param metrics.image.tag Redis&reg; exporter image tag
  ## @param metrics.image.digest Redis&reg; exporter image digest in the way sha256:aa.... Please note this parameter, if set, will override the tag
  ## @param metrics.image.pullPolicy Redis&reg; exporter image pull policy
  ## @param metrics.image.pullSecrets Specify docker-registry secret names as an array
  ##
  image:
    registry: docker.io
    repository: bitnami/redis-exporter
    tag: 1.44.0-debian-11-r16
    digest: ""
    pullPolicy: IfNotPresent
    ## Optionally specify an array of imagePullSecrets.
    ## Secrets must be manually created in the namespace.
    ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
    ## e.g:
    ## pullSecrets:
    ##   - myRegistryKeySecretName
    ##
    pullSecrets: []
  ## @param metrics.resources Metrics exporter resource requests and limits
  ## ref: https://kubernetes.io/docs/user-guide/compute-resources/
  ##
  resources: {}
  ## @param metrics.extraArgs Extra arguments for the binary; possible values [here](https://github.com/oliver006/redis_exporter
  ## extraArgs:
  ##   check-keys: myKey,myOtherKey
  ##
  extraArgs: {}
  ## @param metrics.podAnnotations [object] Additional annotations for Metrics exporter pod
  ##
  podAnnotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "9121"
  ## @param metrics.podLabels Additional labels for Metrics exporter pod
  ##
  podLabels: {}
  ## Containers' Security Context - All fields other than `enabled` get added to the metrics container's security context
  ## @param metrics.containerSecurityContext.enabled Enable Metrics Containers' Security Context
  ## @param metrics.containerSecurityContext.allowPrivilegeEscalation Allow Privilege Escalation for metrics container
  ##
  containerSecurityContext:
    enabled: false
    allowPrivilegeEscalation: false
  ## Enable this if you're using https://github.com/coreos/prometheus-operator
  ##
  serviceMonitor:
    ## @param metrics.serviceMonitor.enabled If `true`, creates a Prometheus Operator ServiceMonitor (also requires `metrics.enabled` to be `true`)
    ##
    enabled: false
    ## @param metrics.serviceMonitor.namespace Optional namespace which Prometheus is running in
    ##
    namespace: ""
    ## @param metrics.serviceMonitor.interval How frequently to scrape metrics (use by default, falling back to Prometheus' default)
    ##
    interval: ""
    ## @param metrics.serviceMonitor.scrapeTimeout Timeout after which the scrape is ended
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#endpoint
    ## e.g:
    ## scrapeTimeout: 10s
    ##
    scrapeTimeout: ""
    ## @param metrics.serviceMonitor.selector Prometheus instance selector labels
    ## ref: https://github.com/bitnami/charts/tree/master/bitnami/prometheus-operator#prometheus-configuration
    ## e.g:
    ## selector:
    ##   prometheus: my-prometheus
    ##
    selector: {}
    ## @param metrics.serviceMonitor.labels ServiceMonitor extra labels
    ##
    labels: {}
    ## @param metrics.serviceMonitor.annotations ServiceMonitor annotations
    ##
    annotations: {}
    ## @param metrics.serviceMonitor.jobLabel The name of the label on the target service to use as the job name in prometheus.
    ##
    jobLabel: ""
    ## @param metrics.serviceMonitor.relabelings RelabelConfigs to apply to samples before scraping
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#relabelconfig
    ##
    relabelings: []
    ## @param metrics.serviceMonitor.metricRelabelings MetricRelabelConfigs to apply to samples before ingestion
    ## ref: https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#relabelconfig
    ##
    metricRelabelings: []
  ## Custom PrometheusRule to be defined
  ## The value is evaluated as a template, so, for example, the value can depend on .Release or .Chart
  ## ref: https://github.com/coreos/prometheus-operator#customresourcedefinitions
  ## @param metrics.prometheusRule.enabled Set this to true to create prometheusRules for Prometheus operator
  ## @param metrics.prometheusRule.additionalLabels Additional labels that can be used so prometheusRules will be discovered by Prometheus
  ## @param metrics.prometheusRule.namespace namespace where prometheusRules resource should be created
  ## @param metrics.prometheusRule.rules Create specified [rules](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/), check values for an example.
  ##
  prometheusRule:
    enabled: false
    additionalLabels: {}
    namespace: ""
    ## These are just examples rules, please adapt them to your needs.
    ## Make sure to constraint the rules to the current postgresql service.
    ##  - alert: RedisDown
    ##    expr: redis_up{service="{{ template "common.names.fullname" . }}-metrics"} == 0
    ##    for: 2m
    ##    labels:
    ##      severity: error
    ##    annotations:
    ##      summary: Redis&reg; instance {{ "{{ $instance }}" }} down
    ##      description: Redis&reg; instance {{ "{{ $instance }}" }} is down.
    ##  - alert: RedisMemoryHigh
    ##    expr: >
    ##       redis_memory_used_bytes{service="{{ template "common.names.fullname" . }}-metrics"} * 100
    ##       /
    ##       redis_memory_max_bytes{service="{{ template "common.names.fullname" . }}-metrics"}
    ##       > 90
    ##    for: 2m
    ##    labels:
    ##      severity: error
    ##    annotations:
    ##      summary: Redis&reg; instance {{ "{{ $instance }}" }} is using too much memory
    ##      description: Redis&reg; instance {{ "{{ $instance }}" }} is using {{ "{{ $value }}" }}% of its available memory.
    ##  - alert: RedisKeyEviction
    ##    expr: increase(redis_evicted_keys_total{service="{{ template "common.names.fullname" . }}-metrics"}[5m]) > 0
    ##    for: 1s
    ##    labels:
    ##      severity: error
    ##    annotations:
    ##      summary: Redis&reg; instance {{ "{{ $instance }}" }} has evicted keys
    ##      description: Redis&reg; instance {{ "{{ $instance }}" }} has evicted {{ "{{ $value }}" }} keys in the last 5 minutes.
    ##
    rules: []
  ## @param metrics.priorityClassName Metrics exporter pod priorityClassName
  ##
  priorityClassName: ""
  ## @param metrics.service.type Kubernetes Service type (redis metrics)
  ## @param metrics.service.loadBalancerIP Use serviceLoadBalancerIP to request a specific static IP, otherwise leave blank
  ## @param metrics.service.annotations Annotations for the services to monitor.
  ## @param metrics.service.labels Additional labels for the metrics service
  ##
  service:
    type: ClusterIP
    ## @param metrics.service.clusterIP Service Cluster IP
    ## e.g.:
    ## clusterIP: None
    ##
    clusterIP: ""
    loadBalancerIP: ""
    annotations: {}
    labels: {}

## @section Sysctl Image parameters
##

## Sysctl InitContainer
## Used to perform sysctl operation to modify Kernel settings (needed sometimes to avoid warnings)
##
sysctlImage:
  ## @param sysctlImage.enabled Enable an init container to modify Kernel settings
  ##
  enabled: false
  ## @param sysctlImage.command sysctlImage command to execute
  ##
  command: []
  ## @param sysctlImage.registry sysctlImage Init container registry
  ## @param sysctlImage.repository sysctlImage Init container repository
  ## @param sysctlImage.tag sysctlImage Init container tag
  ## @param sysctlImage.digest sysctlImage Init container digest in the way sha256:aa.... Please note this parameter, if set, will override the tag
  ## @param sysctlImage.pullPolicy sysctlImage Init container pull policy
  ## @param sysctlImage.pullSecrets Specify docker-registry secret names as an array
  ##
  registry: docker.io
  repository: bitnami/bitnami-shell
  tag: 11-debian-11-r40
  digest: ""
  pullPolicy: IfNotPresent
  ## Optionally specify an array of imagePullSecrets.
  ## Secrets must be manually created in the namespace.
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ## e.g:
  ## pullSecrets:
  ##   - myRegistryKeySecretName
  ##
  pullSecrets: []
  ## @param sysctlImage.mountHostSys Mount the host `/sys` folder to `/host-sys`
  ##
  mountHostSys: false
  ## Container resource requests and limits
  ## ref: https://kubernetes.io/docs/user-guide/compute-resources/
  ## @param sysctlImage.resources.limits The resources limits for the container
  ## @param sysctlImage.resources.requests The requested resources for the container
  ##
  resources:
    ## Example:
    ## limits:
    ##    cpu: 100m
    ##    memory: 128Mi
    ##
    limits: {}
    ## Examples:
    ## requests:
    ##    cpu: 100m
    ##    memory: 128Mi
    ##
    requests: {}
javsalgar commented 1 year ago

Hi,

Just a note to let you know that I was able to reproduce the issue. I will open a task for further investigation. Thanks for reporting it.

javsalgar commented 1 year ago

Hi,

I tried again, and it turned out that the networking in my cluster was broken. I tried recreating a minikube cluster and worked without issues

$ helm install rd bitnami/redis-cluster
NAME: rd
LAST DEPLOYED: Wed Oct 19 11:38:33 2022
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis-cluster
CHART VERSION: 8.2.4
APP VERSION: 7.0.5** Please be patient while the chart is being deployed **

To get your password run:
    export REDIS_PASSWORD=$(kubectl get secret --namespace "default" rd-redis-cluster -o jsonpath="{.data.redis-password}" | base64 -d)

You have deployed a Redis&reg; Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis&reg; cluster:

1. Run a Redis&reg; pod that you can use as a client:
kubectl run --namespace default rd-redis-cluster-client --rm --tty -i --restart='Never' \
 --env REDIS_PASSWORD=$REDIS_PASSWORD \
--image docker.io/bitnami/redis-cluster:7.0.5-debian-11-r0 -- bash

2. Connect using the Redis&reg; CLI:

redis-cli -c -h rd-redis-cluster -a $REDIS_PASSWORD

/tmp/apps/appsmith/deploy/helm

$ kubectl get pods
NAME                 READY   STATUS    RESTARTS      AGE
rd-redis-cluster-0   1/1     Running   0             112s
rd-redis-cluster-1   1/1     Running   1 (37s ago)   112s
rd-redis-cluster-2   1/1     Running   1 (40s ago)   112s
rd-redis-cluster-3   1/1     Running   1 (35s ago)   112s
rd-redis-cluster-4   1/1     Running   1 (39s ago)   112s
rd-redis-cluster-5   1/1     Running   1 (38s ago)   112s
paulf-git commented 1 year ago

I have the same issue when running an EKS 1.23 cluster. EKS cluster running 1.20 has no issues. Same logs as @baptiste-gaillet. Nodes sit at READY 0/1 state. Definitely an issue in EKS deployments vs minikube deployments.

javsalgar commented 1 year ago

This is strange, apart from the version, is there anything different in the configuration?

baptiste-gaillet commented 1 year ago

Hi.

Only change :

javsalgar commented 1 year ago

This is weird, we test our charts with GKE and do not see this issue. Could you add more details on the cluster configuration you created in GKE?

baptiste-gaillet commented 1 year ago

Hello @javsalgar , it was a standard config, without special options.

But I just see this : https://github.com/bitnami/charts/issues/10172

Somone else post the same probleme on GKE on the closed issue 3 days ago.

anessi commented 1 year ago

I had the same problem on K8S 1.23.8 (Rancher) with an NFS storage (Not sure if that's relevant. NFS is probably not the best choice anyway, but it's a test cluster.). Two Redis Clusters in two different namespaces on the same K8S cluster were behaving differently. One was ok, the other one would not start with the same issue described here.

After deleting the PVCs the cluster was starting up again (same config as when it was broken). That's more like a workaround of course but it might help with the analysis. Let me know if you need more details.

javsalgar commented 1 year ago

Hi,

So it seems that there was data persistence from a previous install? Or could it be that the configuration got corrupted?

mfamador commented 1 year ago

I am having the same issue with AKS 1.23.8. One previous installation (older kubernetes), the same image version, on a different namespace is running well.

anessi commented 1 year ago

@javsalgar : there was for sure some existing data on the PVC from a previous install. Unfortunately there's no indication in the logs (e.g. an error message or similar) that would tell that something is not working. I also enabled the debug flag in the chart and re-deployed, but that does not show additional hints why it does not work.

We would need some more hints what specifically to check when this problem happens again.

CharlieC3 commented 1 year ago

I found a temporary and simple workaround for now which may help others.

For some context, it seems the first redis pod in the cluster (redis-0) is designated as the pod responsible for creating the cluster judging from the STS's args parameter. However when it attempts to do this, the other pods in the STS either don't exist yet or are in the middle of booting up and cannot accept connections yet. So, pod redis-0 silently fails to create the cluster. Simply restarting redis-0 is not enough to remedy the situation.

Instead, I had to delete the PVC & PV of the redis-0 pod and restart it. When it came back up with a fresh disk, it re-attempted to create the cluster and it worked just fine because the other redis pods were already running and ready to accept connections. The readiness probes for all redis pods turned healthy in a few seconds. Not sure if this matters, but I'm using an Istio service mesh in my cluster.

Regarding a long-term solution, ideally two things would change:

  1. The pod responsible for creating the cluster would emit an error log if the cluster creation failed, or if the pods needed to be available to create the cluster are not available.
  2. The pod responsible for creating the cluster would re-check the above condition every few seconds, and retry creating the cluster if the previous attempt failed.
mfamador commented 1 year ago

Wow! Nice catch, @CharlieC3 ! Mine is also working now! Thanks a lot!

baptiste-gaillet commented 1 year ago

Good job, so actually, the charts is not production ready with this, too much risk ;p

javsalgar commented 1 year ago

Hi @CharlieC3, thank you so much for spotting the issue. We will work on fixing the issue. As you spotted the problem, would you like to submit a PR in https://github.com/bitnami/containers/tree/main/bitnami/redis-cluster?

github-actions[bot] commented 1 year ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 1 year ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

CharlesB2 commented 1 year ago

Hello @javsalgar are there updates on the ongoing fix for this issue?

jeicopercy commented 1 year ago

I had the same problem but my cluster is running K8S 1.20, for fixing the problem I use the same version that is work fine in other cluster with k8s 1.20, I used helm chat redis-cluster 8.2.7 and it's works ok!

In my case I had first other problem with Redis but with its volumen, then, trying fix it I reinstalled Redis but never work fine with version latest, was necessary return to helm chart v8.2.7.

I hope that other user can test this same solution and confirm us if work fine too.

deshah04 commented 1 year ago

I have the same problem but my cluster is running K8S 1.20, I used helm chat redis-cluster 8.2.7, but the issue is not resolved. This problem occurs intermittently. When I create a new cluster, it works sometimes. And when it doesn't work, it reports "Readiness probe failed: cluster_state:fail" on redis-0.

Any idea when this will be fixed?

javsalgar commented 1 year ago

Hi,

Are you using the default values? Just to understand where the root of the issue is.

github-actions[bot] commented 1 year ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

zek commented 1 year ago

Hi,

Are you using the default values? Just to understand where the root of the issue is.

This is my config. Sometimes it works sometimes not.

  master:
    persistence:
      enabled: true
  replica:
    replicaCount: 3
    persistence:
      enabled: true
  sentinel:
    enabled: true

Disabling persistence is not working also.

Pod logs

20:49:39.82 INFO ==> about to run the command: REDISCLI_AUTH=$REDIS_PASSWORD timeout 220 redis-cli -h app-db-production-redis.app.svc.cluster.local -p 26379 sentinel get-master-addr-by-name mymaster

zek commented 1 year ago 
 21:05:20.13 INFO  ==> about to run the command: REDISCLI_AUTH=$REDIS_PASSWORD timeout 220 redis-cli -h app-db-production-redis.app.svc.cluster.local -p 26379 sentinel get-master-addr-by-name mymaster
 21:09:00.14 INFO  ==> Configuring the node as master
1:C 05 Mar 2023 21:09:00.156 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 05 Mar 2023 21:09:00.156 # Redis version=7.0.9, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 05 Mar 2023 21:09:00.156 # Configuration loaded
1:M 05 Mar 2023 21:09:00.157 * monotonic clock: POSIX clock_gettime
1:M 05 Mar 2023 21:09:00.158 * Running mode=standalone, port=6379.
1:M 05 Mar 2023 21:09:00.158 # Server initialized
1:M 05 Mar 2023 21:09:00.160 * Creating AOF base file appendonly.aof.1.base.rdb on server start
1:M 05 Mar 2023 21:09:00.162 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
1:M 05 Mar 2023 21:09:00.162 * Ready to accept connections

I see that "Configuring the node as master" takes 4 minutes. So all we have to do is wait 4-5 minutes. IDK why it takes this much time.

javsalgar commented 1 year ago

I imagine that it has to do with all the nodes getting scheduled and ready. Is the statefulset deploying the nodes in order?

defyjoy commented 1 year ago

I found a temporary and simple workaround for now which may help others.

For some context, it seems the first redis pod in the cluster (redis-0) is designated as the pod responsible for creating the cluster judging from the STS's args parameter. However when it attempts to do this, the other pods in the STS either don't exist yet or are in the middle of booting up and cannot accept connections yet. So, pod redis-0 silently fails to create the cluster. Simply restarting redis-0 is not enough to remedy the situation.

Instead, I had to delete the PVC & PV of the redis-0 pod and restart it. When it came back up with a fresh disk, it re-attempted to create the cluster and it worked just fine because the other redis pods were already running and ready to accept connections. The readiness probes for all redis pods turned healthy in a few seconds. Not sure if this matters, but I'm using an Istio service mesh in my cluster.

Regarding a long-term solution, ideally two things would change:

1. The pod responsible for creating the cluster would emit an error log if the cluster creation failed, or if the pods needed to be available to create the cluster are not available.

2. The pod responsible for creating the cluster would re-check the above condition every few seconds, and retry creating the cluster if the previous attempt failed.

This is a solution but the fix is still not ready . I am still facing issue with redis cluster v8.3.10 . Kubernetes Server Version: v1.24.10-eks-48e63af

image

image

marcosbc commented 1 year ago

While I don't know what could be causing this issue, it does seem to work on our environments.

What you can do is to enable diagnostics mode (--set diagnosticMode.enabled=true) and enter the pod, start the service like the chart would, and try to find the issue. While it requires some troubleshooting on your side, you will have shell access that should aid in identifying the issue.

ikersuen commented 1 year ago

I got similar issue with:

helm-chart version 7.6.4 redis image version 6.2.7 (cannot use redis 7 since due to metrics issue https://github.com/RedisGrafana/grafana-redis-datasource/issues/271) kubernetes version 1.23.8 istio-service mesh sidecar enabled redis-metrics enabled

redis-0 is popping out and showing "I/O error Node redis-3.redis-3-headless not ready, waiting for all the nodes to be ready" and all pods keep in the state of "2/3 Running"

After waited for around 1 hr it suddenly resume normal, or use the workaround remove PV and PVC, restart the pod of redis-0 also worked for my case. Any bugfix will be done on this issue?

marcosbc commented 1 year ago

@ikersuen At this moment, we don't know what is the issue, unfortunately. Also, we are not able to reproduce it on our side either.

If anyone is able to reproduce this issue and troubleshoot the potential issue, we encourage to send a PR with the fix.

nickkalachev commented 1 year ago

I had the issue. Turns out that on the first run I defined the wrong number of nodes. And when I specified the right number I still got "cluster failed" issue. What helped me is to delete all PVCs, delete the chart and install again. Also I have restricted access to cluster, maybe who has the full access doesn't encounter the problem.

defyjoy commented 1 year ago

So here is the thing . What I did was to delete the redis-data-redis-cluster-0 pvc and also deleted the redis-cluster-0 and ArgoCD recreated them . But this is 100% true the first time the cluster failed with the health check failure for sure . the issue still remains . And unfortunately I have no clue why and what is causing the health check to fail even with debug: true here is my values.yaml - 

image:
  debug: true
global:
  storageClass: "efs-redis" 
serviceAccount:
  create: true  
  annotations: 
    eks.amazonaws.com/role-arn: arn:aws:iam::[secret]:role/sfs-csi-driver-role
  automountServiceAccountToken: true
persistence:
  enabled: true  
  path: /bitnami/redis/data
  storageClass: "efs-redis"
  annotations: {}
  accessModes:
    - ReadWriteOnce
  size: 20Gi
cluster:
  nodes: 6
  replicas: 2
  update:    
    addNodes: true    
    currentNumberOfNodes: 6
    currentNumberOfReplicas: 2
existingSecret: "redis-secret"
existingSecretPasswordKey: "password"
usePassword: true
metrics:  
  enabled: true
  serviceMonitor:    
    enabled: false
    interval: "30s"

image

image

Update :

Just for info I am using shared volume ( EFS Storage ) if that has got something to do with this . Instead of EBS .

So now I made few updates and now the cluster will not recover anymore even if I delete the pvc redis-data-redis-cluster-0 and the pod - redis-cluster-0

ikersuen commented 1 year ago

I had the issue. Turns out that on the first run I defined the wrong number of nodes. And when I specified the right number I still got "cluster failed" issue. What helped me is to delete all PVCs, delete the chart and install again. Also I have restricted access to cluster, maybe who has the full access doesn't encounter the problem.

It's hard to maintain for manually deleting all PVCs and reinstall again as a workaround when we came to deployment pipeline.

Another side-effect is sometimes when the cluster successfully created the first time and we deleted the pvc, redis-data-redis-cluster-0 redis node will failed and cannot rejoin cluster. In this case we need to delete the whole cluster and recreate again. Therefore, we cannot just close our eyes and delete the pvc and restart for this issue.

marcosbc commented 1 year ago

Hi everyone. I'd encourage you to enable diagnostics mode if you have a non-working cluster. That will allow you to manually start Redis services and try to troubleshoot the issues.

In addition, do you have a set of clear step-by-step to reproduce this issue? We have not yet been able to reproduce it on our side, and given that it is affecting more and more users, I'm considering creating an internal task. However, before we can do that, we would at least want some extra details on how to reproduce this issue.

github-actions[bot] commented 1 year ago

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

github-actions[bot] commented 1 year ago

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

mandrean commented 1 year ago

Also experiencing this

  1. create fresh kind cluster (v0.17.0, k8s v1.25.3)
  2. deploy redis-cluster chart (v7.6.4) using helm (v3.11.2) and image.tag=6.2 value
  3. kubectl get pods
redis-cluster-0                             1/1     Running             0          2m23s
redis-cluster-1                             0/1     Running             0          2m23s
redis-cluster-2                             0/1     Running             0          2m23s

And then eventually it fails because of too many probe retries:

  1. kubectl describe pod redis-cluster-1 
    Warning  Unhealthy  1s (x7 over 31s)  kubelet            Readiness probe failed: cluster_state:fail

My super dirty workaround for now is to set cluster.nodes=1, but that's obviously not what I was looking for when picking the -cluster chart.

fmulero commented 1 year ago

Hi @mandrean, thanks for using bitnami/charts.

Are you seeing any log like this one in the pod logs?

*** ERROR: Invalid configuration for cluster creation.
*** Redis Cluster requires at least 3 master nodes.
*** This is not possible with 3 nodes and 1 replicas per node.
*** At least 6 nodes are required.
The cluster was already created, the nodes should have recovered it
redis-server "${ARGS[@]}"

Could you share your values and the logs from the pods?

mandrean commented 1 year ago

logs:

❯ k logs redis-cluster-0
redis-cluster 00:50:30.86 
redis-cluster 00:50:30.86 Welcome to the Bitnami redis-cluster container
redis-cluster 00:50:30.86 Subscribe to project updates by watching https://github.com/bitnami/containers
redis-cluster 00:50:30.86 Submit issues and feature requests at https://github.com/bitnami/containers/issues
redis-cluster 00:50:30.86 
redis-cluster 00:50:30.86 INFO  ==> ** Starting Redis setup **
redis-cluster 00:50:30.87 WARN  ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
redis-cluster 00:50:30.87 INFO  ==> Initializing Redis
redis-cluster 00:50:30.87 INFO  ==> Setting Redis config file
Changing old IP 127.1.27.16 by the new one 10.244.0.74
Changing old IP 127.1.27.10 by the new one 10.244.0.70
Changing old IP 127.1.27.11 by the new one 10.244.0.71
Changing old IP 10.244.0.75 by the new one 10.244.0.75
Changing old IP 10.244.0.72 by the new one 10.244.0.72
Changing old IP 10.244.0.73 by the new one 10.244.0.73
redis-cluster 00:50:30.92 INFO  ==> ** Redis setup finished! **

1:C 08 Jun 2023 00:50:30.937 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 08 Jun 2023 00:50:30.937 # Redis version=6.2.12, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 08 Jun 2023 00:50:30.937 # Configuration loaded
1:M 08 Jun 2023 00:50:30.937 * monotonic clock: POSIX clock_gettime
1:M 08 Jun 2023 00:50:30.938 * Node configuration loaded, I'm 1e6cd85603c11c4954a747c9ba5fd6b2b1a7b4c7
                _._                                                  
           _.-``__ ''-._                                             
      _.-``    `.  `_.  ''-._           Redis 6.2.12 (00000000/0) 64 bit
  .-`` .-```.  ```\/    _.,_ ''-._                                  
 (    '      ,       .-`  | `,    )     Running in cluster mode
 |`-._`-...-` __...-.``-._|'` _.-'|     Port: 6379
 |    `-._   `._    /     _.-'    |     PID: 1
  `-._    `-._  `-./  _.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |           https://redis.io       
  `-._    `-._`-.__.-'_.-'    _.-'                                   
 |`-._`-._    `-.__.-'    _.-'_.-'|                                  
 |    `-._`-._        _.-'_.-'    |                                  
  `-._    `-._`-.__.-'_.-'    _.-'                                   
      `-._    `-.__.-'    _.-'                                       
          `-._        _.-'                                           
              `-.__.-'                                               

1:M 08 Jun 2023 00:50:30.938 # Server initialized
1:M 08 Jun 2023 00:50:30.938 * Ready to accept connections
❯ k get pods
redis-cluster-0                                 0/1     Running   1 (97s ago)    2m38s
redis-cluster-1                                 0/1     Running   1 (101s ago)   2m38s
redis-cluster-2                                 0/1     Running   1 (100s ago)   2m38s
redis-cluster-3                                 0/1     Running   2 (2m5s ago)   2m38s
redis-cluster-4                                 0/1     Running   1 (99s ago)    2m38s
redis-cluster-5                                 0/1     Running   1 (98s ago)    2m38s

values:

image:
  tag: 6.2

usePassword: false
persistance: false # also tried without setting it to false

using redis-cluster chart version 7.6.4

anessi commented 1 year ago

@mandrean I think you're hitting issue #15075. The issue here is different (e.g. also happens with persistence enabled). See this comment for more details. Did you try with persistence enabled?

mandrean commented 1 year ago

@anessi yeah! Same with and without persistance

Note that I even included a comment about it in my values example above: image

What worked for me however was to bump the chart version a little bit higher to 8.1.1 I think, instead of 7.6.4?

fmulero commented 1 year ago

Sorry Sebastian, but I am not able to reproduce your issue.

This is my environment and the steps I followed:

$ kind --version
kind version 0.17.0
$ kind create cluster --image kindest/node:v1.25.3
...
$ k version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.25.2
Kustomize Version: v4.5.7
Server Version: v1.25.3
$ helm version
version.BuildInfo{Version:"v3.10.0", GitCommit:"ce66412a723e4d89555dc67217607c6579ffcb21", GitTreeState:"clean", GoVersion:"go1.19.1"}
$ helm install redis-cluster bitnami/redis-cluster --version 7.6.4 --set usePassword=false --set persistence.enabled=false --set image.tag=6.2 --set image.debug=true
NAME: redis-cluster
LAST DEPLOYED: Mon Jun 12 11:51:09 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: redis-cluster
CHART VERSION: 7.6.4
APP VERSION: 6.2.7** Please be patient while the chart is being deployed **

You have deployed a Redis&reg; Cluster accessible only from within you Kubernetes Cluster.INFO: The Job to create the cluster will be created.To connect to your Redis&reg; cluster:

1. Run a Redis&reg; pod that you can use as a client:
kubectl run --namespace default redis-cluster-client --rm --tty -i --restart='Never' \

--image docker.io/bitnami/redis-cluster:6.2 -- bash

2. Connect using the Redis&reg; CLI:

redis-cli -c -h redis-cluster

WARNING: Rolling tag detected (bitnami/redis-cluster:6.2), please note that it is strongly recommended to avoid using rolling tags in a production environment.
+info https://docs.bitnami.com/containers/how-to/understand-rolling-tags-containers/

$ helm get values redis-cluster
USER-SUPPLIED VALUES:
image:
  debug: true
  tag: "6.2"
persistence:
  enabled: false
usePassword: false
$ k get po
NAME              READY   STATUS    RESTARTS        AGE
redis-cluster-0   1/1     Running   1 (7m50s ago)   8m31s
redis-cluster-1   1/1     Running   1 (7m23s ago)   8m31s
redis-cluster-2   1/1     Running   1 (7m22s ago)   8m31s
redis-cluster-3   1/1     Running   1 (7m25s ago)   8m31s
redis-cluster-4   1/1     Running   1 (7m19s ago)   8m31s
redis-cluster-5   1/1     Running   1 (7m19s ago)   8m31s

We can see a restart in all of the pods but after that restart everything seems to be working fine. All restarts appeared because the pods didn't pass the liveness check:

$ k describe po redis-cluster-0
...
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  3m2s                   default-scheduler  Successfully assigned default/redis-cluster-0 to kind-control-plane
  Warning  Unhealthy  2m29s (x5 over 2m49s)  kubelet            Liveness probe failed: Could not connect to Redis at localhost:6379: Connection refused
  Normal   Killing    2m29s                  kubelet            Container redis-cluster failed liveness probe, will be restarted
  Warning  Unhealthy  2m24s (x7 over 2m49s)  kubelet            Readiness probe failed: Could not connect to Redis at localhost:6379: Connection refused
  Warning  Unhealthy  2m19s                  kubelet            Readiness probe failed: cluster_state:fail
  Normal   Pulled     119s (x2 over 2m59s)   kubelet            Container image "docker.io/bitnami/redis-cluster:6.2" already present on machine
  Normal   Created    119s (x2 over 2m59s)   kubelet            Created container redis-cluster
  Normal   Started    119s (x2 over 2m59s)   kubelet            Started container redis-cluster
$ kubectl run --namespace default redis-cluster-client --rm --tty -i --restart='Never' --image docker.io/bitnami/redis-cluster:6.2 bash
If you don't see a command prompt, try pressing enter.
I have no name!@redis-cluster-client:/$ redis-cli -c -h redis-cluster
redis-cluster:6379> CLUSTER info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:1016
cluster_stats_messages_pong_sent:1022
cluster_stats_messages_sent:2038
cluster_stats_messages_ping_received:1022
cluster_stats_messages_pong_received:1016
cluster_stats_messages_received:2038

Are you removing the pods or upgrading the cluster as described in #15075? Please keep in mind the limitations of running the cluster without persistence.

mandrean commented 1 year ago

We can see a restart in all of the pods but after that restart everything seems to be working fine

It's a bit flaky for me; sometimes it gets stuck in restart loops?

All restarts appeared because the pods didn't pass the liveness check

Yes, that was clear since before. I think this comment explains it well: https://github.com/bitnami/charts/issues/12901#issuecomment-1306040556

Please keep in mind the limitations of running the cluster without persistence.

We're only using this for a local dev environment using kind, so don't really need persistence tbh!

Ideally the chart would reliably install & always get healthy on first attempt (like most other charts), without any weird behavior like getting stuck/restarts etc.