Different RedisFailover's sentinels join together

[tparsa]

Environment

How are the pieces configured?

• Redis Operator version v1.2.2
• Kubernetes version v1.23.13
• Kubernetes configuration used (eg: Is RBAC active?)

  affinity: {}
  annotations: {}
  container:
  port: 9710
  fullnameOverride: ""
  image:
  pullPolicy: IfNotPresent
  repository: quay.io/spotahome/redis-operator
  tag: v1.2.2
  imageCredentials:
  create: false
  email: someone@example.com
  existsSecrets:
  - registrysecret
  password: somepassword
  registry: url.private.registry
  username: someone
  monitoring:
  enabled: false
  prometheus:
  name: unknown
  serviceAnnotations: {}
  serviceMonitor: false
  nameOverride: ""
  nodeSelector: {}
  replicas: 3
  resources:
  limits:
  cpu: 100m
  memory: 128Mi
  requests:
  cpu: 100m
  memory: 128Mi
  securityContext:
  readOnlyRootFilesystem: true
  runAsNonRoot: true
  runAsUser: 1000
  service:
  port: 9710
  type: ClusterIP
  serviceAccount:
  annotations: {}
  create: true
  name: ""
  tolerations: []
  updateStrategy:
  type: RollingUpdate

[dbackeus]

I think a good solution for this would be to migrate from using IP-addresses to hostnames for both Sentinel's and Redis instances. Ie. use replica-announce-ip <hostname> for Redis instances and sentinel announce-ip <hostname> for Sentinel instances. This would imply starting to use StatefulSet for running sentinels as well.

Hostname support is documented at: https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/#ip-addresses-and-dns-names

The issues around environments which swap IP-addresses around is sort of documented at https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/#sentinel-docker-nat-and-possible-issues. But it focuses on Docker environments rather than Kubernetes.

We are currently working on a DIY helm template hostname based approach to replace this operator after having a critical data loss in our production environment due to Sentinels getting mixed up and telling random Redis instances from different clusters to start following each other.

[githubixx]

I can recommend Bitnami Redis Helm chart: https://artifacthub.io/packages/helm/bitnami/redis It also supports Redis Sentinel. It works very well for us with 20+ Redis Sentinel deployments in a Kubernetes cluster. The chart handles various issues you can face with Redis Sentinel very well. They've quite a lot Helm charts for other services too: https://github.com/bitnami/charts/tree/main/bitnami

[sapisuper]

@githubixx correct better using Bitnami Redis. I got issue in OT-REDIS-OPERATOR regarding failover. the failover doesn't running smoothly.