add: taints for machinePools

[VanillaSpoon]

Issue link

What changes have been made

Taints are being implemented to manage the dispatch of appwrappers. When an appwrapper is deployed, it's crucial that its specified nodes are directed to the appropriate scaled-up instances. This mechanism is particularly vital in team-based clusters that scale up multiple instances for various appwrappers. The currrent workflow potentially allows appwrappers to distribute their workload across scaled-up instances not assigned to them, leading to resource mixups.

By applying taints, we ensure that pods are assigned to the correct nodes, preventing such issues.

The same will be done for nodepools separately.

Verification steps

Deploy an appwrapper with instascale enabled, and specified instances. View you scaled up machinepool in the dashboard, and check that a taint has been applied with <aw-name> = value1:PreferNoSchedule.

Once the nodes are scaled you can again check within compute and nodes. Feel free to filter with labels=aw-name=aw-name. When viewing each nodes Details you will see again the same taint is visible.

Ensure the appwrappers pods are deploying successfully to these nodes.

To ensure the opposite, feel free to scale up resources with instascale, and deploy a separate appwrapper, ensuring this appwrappers pods are not deployed to the scaled up resources.

Checks

• [ ] I've made sure the tests are passing.
• Testing Strategy
  • [ ] Unit tests
  • [ ] Manual tests
  • [ ] Testing is not required for this change

[openshift-ci[bot]]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please assign maxusmusti for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[OWNERS](https://github.com/project-codeflare/instascale/blob/main/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[asm582]

Thanks for this PR, please make sure we patch taints and change tolerations in the reuse scenario.

[VanillaSpoon]

Thanks @asm582 There are currently no reuse scenarios in place for machinepools

[asm582]

Thanks @asm582 There are currently no reuse scenarios in place for machinepools I think reuse is important. we should have it as a part of this PR or the following PR. Can you please check if an issue exists and if not please create a new one.

[VanillaSpoon]

Thanks @asm582 There are currently no reuse scenarios in place for machinepools I think reuse is important. we should have it as a part of this PR or the following PR. Can you please check if an issue exists and if not please create a new one.

Hey, I've spent some time looking into it. Great idea, I'll open a new issue and get reuse added for each machine type :)