Open alexec opened 4 years ago
Note that the Kubeflow Pipelines team does not require this feature to be present in Argo (KFP works fine as it is). But this might be a neat feature for people using raw Argo.
Here is an example of using this feature in KF Pipelines: https://github.com/kubeflow/pipelines/blob/master/sdk/python/tests/components/test_data/retail_product_stockout_prediction_pipeline.component.yaml
Here is an example of graph (DAG) using task that references a component (template) using an HTTPS URL.
...
implementation:
graph:
tasks:
Automl split dataset table column names:
componentRef:
url: https://raw.githubusercontent.com/kubeflow/pipelines/b3179d86b239a08bf4884b50dbf3a9151da96d66/components/gcp/automl/split_dataset_table_column_names/component.yaml
arguments:
dataset_path:
taskOutput:
outputName: dataset_path
taskId: Automl import data from bigquery
target_column_name:
graphInput:
inputName: target_column_name
table_index: '0'
Here is the schema of the ComponentReference used by KFP components.
Thank you @Ark-kun. I've created this issue and we'll see how many up-votes it get. I can imagine people might like this - I'm intrigued how we can verify URLs.
Plus one for this. We also have a similar issue. https://github.com/argoproj/argo/issues/2900
We could maybe easily address this using the artifact code we already have.
@alexec can you give more hints on using the artifact code? do you mean that we can use the executor of the artifact to support URL loading? thanks
No. The real problem here is not rendering templates, but accessing the source in a secure and configurable manner. As we already have code for reading and writing artifacts to various locations - we could reuse that code here.
Got you, I can look at this. Thanks Alex.
On Mon, May 4, 2020 at 6:45 PM Alex Collins notifications@github.com wrote:
No. The real problem here is not rendering templates, but accessing the source in a secure and configurable manner. As we already have code for reading and writing artifacts to various locations - we could reuse that code here.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/argoproj/argo/issues/2847#issuecomment-623794599, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAK5R6O4WLDASCUOIK3T3NLRP5VURANCNFSM4MRLZ2JA .
Summary
Allow resolution of templates by using URLs.
Motivation
So we can have templates in a single place.
Proposal
This is a feature that Kubeflow has and @Ark-kun mentions in #2793.
@mukulikak @jessesuen
Message from the maintainers:
If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.