[feature] Parametrize launcher image in KFPV2

kubeflow / pipelines

Machine Learning Pipelines for Kubeflow

https://www.kubeflow.org/docs/components/pipelines/

Apache License 2.0

3.55k stars 1.6k forks source link

[feature] Parametrize launcher image in KFPV2 #10229

Closed rimolive closed 9 months ago

rimolive commented 10 months ago

Feature Area

/area backend

What feature would you like to see?

Launcher image could be parametrized in a way that users can make use of the following tasks:

Work on a custom launcher image for additional checks before running a pipeline step
Run Kubeflow Pipelines in a disconnected environment

What is the use case or pain point?

Launcher image is hard-coded, and this is bad practice IMHO. We should be able to let users parameterize any image used by KFP, and launcher image is that example.

Is there a workaround currently?

There is no possible workarounds

Love this idea? Give it a 👍.

nirajkl commented 10 months ago

Hi @rimolive ,

please suggest on timelines when the changes will be implemented.

thanks.

and-1 commented 9 months ago

It would be great If you parametrize driver image also

rimolive commented 9 months ago

@nirajkl @and-1 Please check latest status. We have an ongoing PR to introduce launcher and driver image parametrization.

chensun commented 9 months ago

This was implemented and released in KFP 2.0.5

thesuperzapper commented 4 months ago

I want to highlight that V2_LAUNCHER_IMAGE will NOT affect V2_COMPATIBLE mode runs.

These runs will still use the _DEFAULT_LAUNCHER_IMAGE which was hard-coded in the version of the kfp SDK you are using which is usually gcr.io/ml-pipeline/kfp-launcher:1.8.7:

It is defined here in kfp==2.7.0
It is defined here in kfp==1.8.7

rimolive commented 4 months ago

v2-compatible mode is no longer supported, and the ideal path is upgrade kfp to the latest release. We're currently in 2.1.0 but we're planning to release 2.2.0 for Kubeflow 1.9.