search

kubeflow / pipelines

Machine Learning Pipelines for Kubeflow
https://www.kubeflow.org/docs/components/pipelines/
Apache License 2.0
3.62k stars 1.63k forks source link

[feature] [SDK] Extract Executor to its own package #11345

Open leseb opened 3 weeks ago

leseb commented 3 weeks ago

Feature Area

/area sdk

What feature would you like to see?

To improve modularity and reduce dependency overhead in Kubeflow Pipelines, I propose extracting the executor component into a standalone package. Currently, the executor is tightly coupled with other SDK components, resulting in unnecessary dependencies during execution. By isolating the executor into its own package, we can minimize the dependency footprint and enhance flexibility for users seeking a more lightweight solution.

Something like kfp.executor package would be nice. Thanks for considering this proposal.

Love this idea? Give it a 👍.

leseb commented 3 weeks ago

@HumairAK FYI

HumairAK commented 2 weeks ago

past attempt on this: https://github.com/kubeflow/pipelines/pull/9710

chensun commented 2 weeks ago

Something like kfp.executor package would be nice. Thanks for considering this proposal.

Second part of this effort was https://github.com/kubeflow/pipelines/pull/9738, which was later reverted by https://github.com/kubeflow/pipelines/pull/9874 due to some technical challenges IIRC. @connor-mccarthy do you recall the exact issue?

HumairAK commented 2 weeks ago

In today's community call it was decided that this approach makes sense and got a +1 to move forward.

chensun commented 2 weeks ago

And what we ended up as a shortcut was to install kfp without any dependencies:

https://github.com/connor-mccarthy/pipelines/blob/fcdff294a6323f6cb1c0e574fc7aa5ccc25e420b/sdk/python/kfp/dsl/component_factory.py#L146-L149

which is arguably the same level of dependency as installing a subpackage like kfp.runtime.

leseb commented 2 weeks ago

Could we get a list of the technical challenges that caused the initial revert of the previous attempt? This will help us avoid repeating the same issues. Thanks!