search

IBM / data-prep-kit

Open source project for data preparation of LLM application builders
https://ibm.github.io/data-prep-kit/
Apache License 2.0
321 stars 135 forks source link

Add Tolerations and node selector to KFP pods #643

Closed revit13 closed 1 month ago

revit13 commented 1 month ago

Why are these changes needed?

Add Tolerations and node selector to KFP pods

To apply kuberenetes Tolerations or nodeSelector to KFP pods, you need to set KFP_TOLERATIONS or KFP_NODE_SELECTOR environment variables respectively before compiling the pipeline. Here's an example:

export KFP_TOLERATIONS='[{"key": "key","operator": "Equal", "value1": "value", "effect": "NoSchedule"}]'

export KFP_NODE_SELECTOR='{"label_key":"cloud.google.com/gke-accelerator","label_value":"nvidia-tesla-p4"}'

Related issue number (if any).

Relates to #620

roytman commented 1 month ago

Can you please combine Tolerance settings with PR #627 , so users will not have to update several parts.

revit13 commented 1 month ago

Can you please combine Tolerance settings with PR #627 , so users will not have to update several parts.

Done for kfp v1, I'll open an issue as you request for kfp v2.