search

armadaproject / armada-operator

Apache License 2.0
13 stars 10 forks source link

redis pods cannot be scheduled #309

Closed wilson-duan closed 3 months ago

wilson-duan commented 4 months ago

I noticed that some of the redis pods never run - specifically, all three replicas of redis-ha-server-2. It is quite easy to diagnose the issue; by running kubectl describe pod redis-ha-server-2, we can see that the redis-ha-server-2 pod cannot be scheduled onto any of the 3 nodes (1 control plane, 2 worker nodes) because it has an antiaffinity with the other redis pods, and it cannot be scheduled on the control plane. Since the 2 worker nodes already host the pods redis-ha-server-0 and redis-ha-server-1, the antiaffinity of the redis-ha-server-2 pod results in the pod never running. This problem is easily solved by introducing a 3rd worker node to the cluster; I installed Armada on a K8s cluster hosted on 4 VMs, and I was able to get all the redis servers working since my cluster has 3 worker nodes.

Thus, I would recommend modifying the kind installation to have 3 worker nodes rather than 2.

dejanzele commented 4 months ago

Hi,

The make kind-all is meant to serve as a Quickstart and not a production setup, I'll try and reduce the config so it starts only 1 Pod to reduce resource requirements as much as possible.