redis-cluster pods not distributing across multiple zones with topology spread constraint enabled for zone

[TANISH-18]

Hi @cin , i am deploying fed-redis-cluster. even after enabling zone level topology i can see that pods of same shards are not spread across multiple zones.

POD NAME IP NODE ID ZONE USED MEMORY MAX MEMORY KEYS SLOTS

• rediscluster-node-for-redis-4b86s 192.168.184.25 10.31.48.73 efba48e4f7426eca137f517f3d99470d92ac1521 smf-zone-c3 3.72M 2.80G db0=79 10924-16383
  | rediscluster-node-for-redis-hxb77 192.168.215.39 10.31.48.89 154c075ffef92f4ac02a8d42dc3753ea2efab97b smf-zone-c1 3.37M 2.80G db0=79
  | rediscluster-node-for-redis-pj4lt 192.168.24.59 10.31.48.71 68f05605dddd5f464efe1cd4888d830aa35658c1 smf-zone-c1 3.34M 2.80G db0=79
• rediscluster-node-for-redis-mnrkt 192.168.138.167 10.31.48.87 50fc7c3df8f41bfd93f8f309b07bd1e353786c0e smf-zone-c0 3.82M 2.80G db0=83 5462-10923
  | rediscluster-node-for-redis-4kmqk 192.168.7.136 10.31.48.76 89c69df5fa0dd1f10407ae0299b9df59c67e688b smf-zone-c3 3.39M 2.80G db0=83
  | rediscluster-node-for-redis-bwh6n 192.168.215.2 10.31.48.89 c44b6fc635a33b1d1755d9eb206283f71ea8a825 smf-zone-c1 3.40M 2.80G db0=83
• rediscluster-node-for-redis-n2s7j 192.168.215.52 10.31.48.89 2f82fc3f4ebff0f10b919545e2f774ea765c11c2 smf-zone-c1 4.11M 2.80G db0=99 0-5461
  | rediscluster-node-for-redis-pn4zf 192.168.3.232 10.31.48.74 5398b6e6fd8f5fc7f3d1e2181901330ae4399a9f smf-zone-c3 3.45M 2.80G db0=99
  | rediscluster-node-for-redis-s5v8l 192.168.34.14 10.31.48.88 6fc8a525a165c6119d29a367a21ea8a09cfb9cd1 smf-zone-c0 3.45M 2.80G db0=100

topologySpreadConstraints:

• labelSelector: matchLabels: app.kubernetes.io/component: database app.kubernetes.io/name: node-for-redis maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway
• labelSelector: matchLabels: app.kubernetes.io/component: database app.kubernetes.io/name: node-for-redis maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: ScheduleAnyway

[TANISH-18]

[TANISH-18]

if you see in the first shard two redis pods are in the same zone ideally it should be in different zone. we have total 4 zone in our cluster and each zone has 3 k8 nodes

[cin]

Thanks for the issue @TANISH-18 and sorry for the delay. This is actually a known issue if you are sharing worker nodes. There's only so much control we have over where the kubernetes scheduler puts pods, so we try to do the best we can. Notice you have one worker with 3 pods scheduled on it and two workers with only one pod scheduled. There's not much we can do to fix that at the operator level. If HA is really your priority, I'd advise against sharing nodes. If a node dies, so do the pods on that worker. So in this case, you'd lose whatever was on the 3 pods on 10.31.48.89.

I actually started looking into some ways to address this and created a PR to investigate -- see #55. So far, what I tried there is only marginally better than the existing logic. The placement logic really needs to be re-written to address this issue, but it gets complicated (see comments on the PR draft). Unfortunately I don't really have the time to address this issue at the moment (due to other work). TBH, I don't really consider this to be a priority because worker nodes shouldn't be shared. I get that you may want to share nodes in non-production environments, but you also don't care about HA as much in those environments.

Sorry I know this is not the answer you wanted to hear, but hopefully understanding the limitation will help you work around it until there is a better solution in place.