kubeflow / pipelines

Machine Learning Pipelines for Kubeflow
https://www.kubeflow.org/docs/components/pipelines/
Apache License 2.0
3.55k stars 1.6k forks source link

How to read kubernetes secrets in v2 pipelines #10528

Closed milind-u closed 6 months ago

milind-u commented 6 months ago

How could one attach kubernetes secrets/configMaps to a pipeline in a multi user setup or fetch secrets from the dsl python code? In https://kfp-kubernetes.readthedocs.io/en/kfp-kubernetes-1.0.0/#secret-as-mounted-volume, there is an example to do this in kubeflow pipelines v1, but is there an equivalent syntax in v2?

rimolive commented 6 months ago

Can you ellaborate? You could post some code snippets to clarify your use case.

milind-u commented 6 months ago

Sure. In kfp v1, you could read a secret in a kubeflow pipeline from kubernetes with the following code:

from kfp import dsl
from kfp import kubernetes

@dsl.component
def print_secret():
    with open('/mnt/my_vol') as f:
        print(f.read())

@dsl.pipeline
def pipeline():
    task = print_secret()
    kubernetes.use_secret_as_volume(task,
                                    secret_name='my-secret',
                                    mount_path='/mnt/my_vol')

Is there an equivalent syntax in kfp v2? There is no longer a kubernetes module in kfp.

rimolive commented 6 months ago

From the link you pointed out: https://kfp-kubernetes.readthedocs.io/en/kfp-kubernetes-1.0.0/#secret-as-mounted-volume:

The kfp-kubernetes package can be installed as a kfp SDK extra dependency with kfp==2.x.x:

pip install kfp[kubernetes] --pre

So what you are doing should be compatible with v2 as well. I'm surprised this code worked with v1, as this Python package was created for v2.

milind-u commented 6 months ago

I see, I was getting confused by the kfp-kubernetes version v1.0.0 and didn't realize this is a different package from kfp. I didn't actually try the above with kfp v1.

Thank you!