search

canonical / bundle-kubeflow

Charmed Kubeflow
Apache License 2.0
104 stars 50 forks source link

Research how to set the labels correctly for v2 kfp pipelines #1169

Closed misohu closed 1 day ago

misohu commented 1 day ago

Context

We want to set needed labels on kfp steps pods in order to propagate spark settings to environment.

What needs to get done

  1. Find the right way to set labels in kfp v2 pipelines

Definition of Done

  1. We know what is the right way to set labels in kfp v2 pipelines
syncronize-issues-to-jira[bot] commented 1 day ago

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6633.

This message was autogenerated

NohaIhab commented 1 day ago

We discovered the kfp-kubernetes python package, it can be installed by running:

pip3 install kfp[kubernetes]

Once installed, it can be used to add a label to a pod in a pipeline as follows:

@dsl.pipeline
def my_pipeline():
    task = comp()
    kubernetes.add_pod_label(
        task,
        label_key='kubeflow.com/kfp',
        label_value='pipeline-node',
    )

We were able to test this from a notebook and verify that it applies the label for spark PodDefault. Note that the versioning scheme of this package is not the same as the kfp sdk, so while the latest version is 1.4.0, it is not outdated. You can see that the latest commit was 3 weeks ago here.