search

elyra-ai / elyra

Elyra extends JupyterLab with an AI centric approach.
https://elyra.readthedocs.io/en/stable/
Apache License 2.0
1.81k stars 339 forks source link

Evaluate use of Kubeflow Pipelines SDK v2 #1930

Open ptitzler opened 3 years ago

ptitzler commented 3 years ago

Is your feature request related to a problem? Please describe. Evaluate whether an upgrade to the KFP SDK v2 provides benefits and should be performed in in Elyra 3.x.

ptitzler commented 2 years ago

SDK v2 is still in beta. Waiting for it to be classified as stable or an urgent need arises that requires a switch.

ozen commented 1 year ago

@ptitzler Does this mean components written using KF pipelines sdk v2 cannot be used in Elyra?

Repo activity suggests KubeFlow 1.6.0 will be released next month with KF Pipelines 2.0.0 which makes SDK v2 default. Currently KF v1.6.0-rc.1 installs KF Pipelines 2.0.0-alpha.3. In Pipelines v2, v1 mode is removed from the namespace: https://github.com/kubeflow/pipelines/issues/7238

ptitzler commented 1 year ago

Elyra uses the SDK to communicate with KF (create experiments, upload pipelines, etc) and compile pipelines. I don't believe any of those tasks would fail after an upgrade. However, the visual pipeline editor parses component specifications to render the properties are used to customize the behavior of pipeline nodes. I do anticipate that some things related to this might not work as expected, but @kiersten-stokes and I can sort those out. I'm temporarily earmarking this for Elyra 4.0, which is a bit out [1], but we might be able to deliver earlier, depending on when a stable KFP SDK is released.

[1] Elyra 4.0 introduces support for JupyterLab 4, which itself has a moving target date.