search

kubeflow / pipelines

Machine Learning Pipelines for Kubeflow
https://www.kubeflow.org/docs/components/pipelines/
Apache License 2.0
3.51k stars 1.58k forks source link

[sdk] Missing Alternative for set_ttl_seconds_after_finished in KFP v2 #10899

Open nasenthi opened 4 weeks ago

nasenthi commented 4 weeks ago

Environment

Steps to reproduce

  1. Create and run any pipeline in KFP v2.
  2. Observe that the pods created by the run are not automatically deleted after the pipeline execution is complete.

In KFP v1, there was a function kfp.dsl.PipelineConf().set_ttl_seconds_after_finished(seconds) that allowed us to delete pods after a specified time. However, in KFP v2, this function has been deprecated. When attempting to use this deprecated function, we encounter errors as documented in the following issues:

Issue - https://github.com/kubeflow/pipelines/issues/10715 Issue - https://github.com/kubeflow/pipelines/issues/9803 Is there any alternative method available in KFP v2 to delete the pods after they have finished executing?

Expected result

There should be a method in KFP v2 SDK to specify the TTLStrategy inside argo workflows to have them cleaned up.

Materials and Reference

Impacted by this bug? Give it a 👍.

gregsheremeta commented 3 weeks ago

I'm working on this

hbelmiro commented 3 weeks ago

/assign @gregsheremeta

papagala commented 3 weeks ago

We are currently migrating all our pipelines to V2.7.0 and experienced the same problem. It was also useful to have a way to add secrets at a pipeline level and image pull secrets. Thanks @gregsheremeta 👍