Azure / azure-sdk-for-python

This repository is for active development of the Azure SDK for Python. For consumers of the SDK we recommend visiting our public developer docs at https://docs.microsoft.com/python/azure/ or our versioned developer docs at https://azure.github.io/azure-sdk-for-python.
MIT License
4.36k stars 2.71k forks source link

Can't set output for pipeline job when loading pipeline component from file #35291

Open skakunm opened 2 months ago

skakunm commented 2 months ago

I have yaml file which specifies a pipeline. I load this pipeline using load_component. Then I convert this component to job by specifying inputs in format pipeline_job = pipeline_component(input1=input1, input2=input2). After that I set outputs of the job using pipeline_job.outputs.output1 = output1. I can see that value of outputs field of pipeline_job object is set correctly, and type of pipeline_job is Pipeline. Unfortunately, the line linked below doesn't use pipeline_job.outputs, but rather pipeline._job_outputs for outputs, thus outputs are not set when job is submitted. https://github.com/Azure/azure-sdk-for-python/blob/32c2e5f9ba9729710210f0c88d33be1f39380bf5/sdk/ml/azure-ai-ml/azure/ai/ml/entities/_builders/pipeline.py#L161 Only place where _job_outputs field is set is in BaseNode constructor, and this is when line pipeline_job = pipeline_component(...) is called, but this doesn't allow passing outputs https://github.com/Azure/azure-sdk-for-python/blob/d5c35788360bcbf3566139cc29a25a8b996f8fd2/sdk/ml/azure-ai-ml/azure/ai/ml/entities/_job/pipeline/_load_component.py#L306 Docstring of Pipeline says that one shouldn't create Pipeline directly but rather using @pipeline decorator, and this doesn't allow setting outputs directly.

Is there a way to set outputs without directly accessing _job_outputs of pipeline_job?

It is strange that this is in example

# example how to change path of output on pipeline level
pipeline_job.outputs.model_output = Output(
    type="uri_folder", mode="rw_mount", path=custom_path
)

here: https://github.com/Azure/azureml-examples/blob/main/sdk/python/jobs/pipelines/1b_pipeline_with_python_function_components/pipeline_with_python_function_components.ipynb

github-actions[bot] commented 2 months ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/azure-ml-sdk @azureml-github.