search

kubeflow / pipelines

Machine Learning Pipelines for Kubeflow
https://www.kubeflow.org/docs/components/pipelines/
Apache License 2.0
3.55k stars 1.6k forks source link

Support inputs with multiple arguments #1933

Closed Ark-kun closed 5 months ago

Ark-kun commented 5 years ago

Need to support variadic inputs in ComponentSpec, Lightweight components, load_component etc.

In command-line we can pass multiple inputs as repeated command-line parameters: --a 1 --a 2. In TaskSpec we can make argument a list: arguments: {a: [1, 2]}. In inputSpec we could add repeated: true or allowMultiple: true property. In ContainerImplementation we could specify the command args as follows: {map: {source: {inputValue: RepInput}, selector: [--a, {item}] }}

Related: https://github.com/kubeflow/pipelines/issues/1901

areshytko commented 4 years ago

is this feature planned to be implemented in the upcoming releases? It would be great to have it in KFP

Ark-kun commented 4 years ago

is this feature planned to be implemented in the upcoming releases? It would be great to have it in KFP

At this moment there we have no bandwidth for implementing this feature. It needs many places to change.

What is the scenario where you might want this feature?

areshytko commented 4 years ago

The use case is the following: we have K-fold a cross-validation task somewhere in the middle of our pipelines, and the number of fold is the parameter to the pipeline. The desire is to distribute computation of each fold to a separate task and run them in parallel since this is a time-consuming operation. Basically this requires the following:

image Task C now requires a variable number of input parameters.

Can we offer our help and contribute to the implementation of this feature under this ticket?

ca-scribner commented 4 years ago

Does anyone have a pattern to make something like this work? The only hacks I've thought of would be maybe something like a simple shell call that takes the values from TaskB-1...TaskB-N and concatenates them to a single string (json or just space deliminated) or maybe to lines in a file, then TaskC takes that concatenation as input and knows how to parse it

Other ideas might be that TaskB- concatenate their results somewhere central and TaskC grabs from there. So maybe TaskB- write results to the same location, and TaskC grep's from there?

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] commented 3 years ago

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

BenoitLeguay commented 3 years ago

Any news on this feature ? :) Thank you !

SaschaHeyer commented 2 years ago

This feature request is closely related to https://github.com/kubeflow/pipelines/issues/3412 Are there any updates on that this is a request I have seen already multiple times.

rimolive commented 5 months ago

Closing this issue. No activity for more than a year.

/close

google-oss-prow[bot] commented 5 months ago

@rimolive: Closing this issue.

In response to [this](https://github.com/kubeflow/pipelines/issues/1933#issuecomment-2039389815): >Closing this issue. No activity for more than a year. > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.