search

IBM / data-prep-kit

Open source project for data preparation of LLM application builders
https://ibm.github.io/data-prep-kit/
Apache License 2.0
307 stars 134 forks source link

[Bug] Ray clusters are created in CI/CD with `imagePullPolicy: Always` #807

Closed revit13 closed 4 days ago

revit13 commented 5 days ago

Search before asking

Component

KFP workflows

What happened + What you expected to happen

Expected Behavior: When creating the Ray cluster, the default ImagePullPolicy should be IfNotPresent. In CI/CD, the transform image is built and loaded into the kind cluster before the test run, ensuring that the local image code is tested. What happen: The Ray cluster was created with ImagePullPolicy set to Always, causing it to always pull the transform image from the remote repository. This resulted in an image pull error when the image was not available in the remote repository, and preventing the local version from being tested.

Reproduction script

run make workflow-test in transforms/universal/noop.

Anything else

No response

OS

Ubuntu

Python

3.10.x

Are you willing to submit a PR?

revit13 commented 4 days ago

closed with https://github.com/IBM/data-prep-kit/pull/808