MCAD doesn't take into account the node tainting

[kpouget]

As part of my Codeflare/MCAD test automation, I observed the following behavior:

mnisttest-user1-head-v9hbd                                        0/1     Pending     0              8m4s    <none>         <none>                         <none>           <none>
nisttest-user1-worker-small-group-mnisttest-user1-hmt8p           0/1     Pending     0              8m4s    <none>         <none>                         <none>           <none>
nisttest-user1-worker-small-group-mnisttest-user1-jhqkm           0/1     Pending     0              8m4s    <none>         <none>                         <none>           <none>

• a RayCluster, part of an AppWrapper, has been instantiated in the cluster, although there's not enough resources to schedule them:

  "message": "0/8 nodes are available: 1 Insufficient cpu, 1 node(s) had untolerated taint {only-test-pods: yes}, 
  3 node(s) didn't match Pod's node affinity/selector, 
  3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. 
  preemption: 0/8 nodes are available: 1 No preemption victims found for incoming pod, 
  7 Preemption is not helpful for scheduling.",

My understanding is that MCAD did not take into account the taint in one of the nodes (1 node(s) had untolerated taint {only-test-pods: yes}) when it decided that the AppWrapper would fit in the cluster. Unfortunately, the only node available for this workload did not have enough CPU to host the RayCluster 1 Insufficient cpu.

---

• the cluster Pods details.
• MCAD logs
• MCAD version: quay.io/project-codeflare/mcad-controller:release-v1.32.0 (found in the Pod details link above)

[kpouget]

AppWrapper state when the RayCluster is pending: appwrapper.yaml.log (from another identical test run)