Two different redis clusters in different namespaces interconnected. Production disturbed.

[PrudhviApx7]

Expected behaviour

Two redis clusters must work seamless if they have different namespace and name. accordign to this question

What do you want to achieve? Run Mulltiple redis clusters in different namespaces with single operator

Actual behaviour

The replicas from cluster/replicated setup have connected to master of another cluster in diffrent namespace.

Describe step by step what you've have done to get to this point

They were working fine from last two months but yesterday at 6am, spot nodes in which these replicas are running have been taken down by GCP and new ones have spinned off during this transition soemthing happened which resulted in sudden drop to 2000+ keys and interconnection of replicas between clusters.

Environment

kubernetes

• Redis Operator version v1.2.4
• Kubernetes version 1.26
• Kubernetes configuration used (eg: Is RBAC active?)

[PrudhviApx7]

My redis failover cluster are names as sync-redis and aurora-redis. In the below logs we can see replicas of sync-redis are made slaves of master of aurora-redis.

[github-actions[bot]]

This issue is stale because it has been open for 45 days with no activity.

[samof76]

This issue is still relevant, and it happens because pods from a different cluster inherit ips that belong to some other cluster and sentinel is still not flushed to look for these new ip addresses. possible fixes

• change the operator to not use mymaster to identify the cluster
• change the sentinel to look up redis statefulset hostnames, rather than using ips. This means that must make the sentinels too statefulset, as all can be host identifiable.
• apply network policies with pod selectors only allowing the pods with the similar selector communicate with pods with labels applied.(no operator changes required)

[samof76]

@PrudhviApx7 Also please avoid using the same port for all tenants.