search

IBM / data-prep-kit

Open source project for data preparation of LLM application builders
https://ibm.github.io/data-prep-kit/
Apache License 2.0
321 stars 135 forks source link

Apply tolerations to ray pods defined by env var in KFP v2 #648

Closed revit13 closed 1 month ago

revit13 commented 1 month ago

Search before asking

Component

KFP workflows

Feature

Related to #620 and PRs https://github.com/IBM/data-prep-kit/pull/643 and https://github.com/IBM/data-prep-kit/pull/627

Following @roytman request in this pr tolerations for ray pods (and kfp pods) can be applied for KFP v1 when exporting the following env var before compiling the pipeline; however, for KFPv2 its is not supported yet.

export KFP_TOLERATIONS='[{"key": "key","operator": "Equal", "value1": "value", "effect": "NoSchedule"}]'

In addition, for both KFP v1 and v2 the tolerations for ray pods can be specified in the pipeline params: ray_head_options and ray_worker_options. for example:

  "cpu": 1,
  "image": "quay.io/dataprep1/data-prep-kit/noop-ray:latest",
  "memory": 4,
  "tolerations": [
    {
      "key": "preprocessing",
      "operator": "Equal",
      "value": "dedicated",
      "effect": "NoSchedule"
    }
  ]
}

Are you willing to submit a PR?

roytman commented 1 month ago

Done with #627 and #643