Create a machine learning pipeline job from YAML file using MLClient

[SiHaoShen]

Is your feature request related to a problem? Please describe.

Currently, developers working with Azure AI ML Python SDK face a significant challenge in setting up complex machine learning pipelines. The process demands writing extensive boilerplate code and specifying numerous configuration parameters manually. This approach not only increases the development time but also introduces a higher risk of errors.

The existing azure ml cli demonstractes a more streamlined approach by using az ml job create --file job.yml that allow developers to create a machine learning command jobs and pipeline jobs through a simple YAML file configuration. However, this functionality is absent in the MLClient library, leading to a less efficient and error-prone workflow.

Describe the solution you'd like

To address this gap, I propose adding an additional argument file_name: Optional[str] to the MLClient.create_or_update()method, enabling it to accept a YAML file as input for creating or updating machine learning pipeline jobs. The YAML file would follow the schema detailed in the Azure Machine Learning Pipeline Job documentation. This approach will simplify the process of setting up complex pipelines by parsing the job configurations directly from a predefined YAML file, thus reducing development time, minimizing the risk of errors, and aligning the MLClient's functionality with the convenience offered by the Azure CLI tool.

Reference:

• https://learn.microsoft.com/en-us/cli/azure/ml/job?view=azure-cli-latest#az-ml-job-create-examples
• https://learn.microsoft.com/en-us/azure/machine-learning/reference-yaml-job-pipeline?view=azureml-api-2
• https://azuresdkdocs.blob.core.windows.net/$web/python/azure-ai-ml/1.15.0/azure.ai.ml.html#azure.ai.ml.MLClient.create_or_update

[github-actions[bot]]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @Azure/azure-ml-sdk @azureml-github.

[kdestin]

It sounds like you might be looking for the load_job function: https://learn.microsoft.com/en-us/python/api/azure-ai-ml/azure.ai.ml?view=azure-python#azure-ai-ml-load-job

from azure.ai.ml import load_job

job = load_job(source="path/to/job.yaml")

You can use the returned job as an argument to ml_client.create_or_update

[SiHaoShen]

I tried this and it works. Thank you for the tip!