search

Azure / azure-cli

Azure Command-Line Interface
MIT License
3.97k stars 2.95k forks source link

StepSequence in v1 no longer available in AzureML v2 #26739

Open newhardwarefound opened 1 year ago

newhardwarefound commented 1 year ago

Related command

az ml job create

Is your feature request related to a problem? Please describe.

I am frustrated when trying to get one AML step to start after another AML step in a pipeline, when there are no input/output files between the 2 steps. It's a useful v1 feature (https://learn.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.stepsequence?view=azure-ml-py) that is removed in v2. Airflow also supports it naturally. Right now I have to modify like 6 different files to add a placeholder parameter for the 2 successive steps.

Describe the solution you'd like

Add a field called prerequisites in the pipelinecommandjob yaml so that i can specify which steps need to complete before starting the current step.

Describe alternatives you've considered

Additional context

yonzhan commented 1 year ago

Thank you for opening this issue, we will look into it.

ghost commented 1 year ago

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @azureml-github.

Issue Details
**Related command** az ml job create **Is your feature request related to a problem? Please describe.** I am frustrated when trying to get one AML step to start after another AML step in a pipeline, when there are no input/output files between the 2 steps. It's a useful v1 feature (https://learn.microsoft.com/en-us/python/api/azureml-pipeline-core/azureml.pipeline.core.stepsequence?view=azure-ml-py) that is removed in v2. Airflow also supports it naturally. Right now I have to modify like 6 different files to add a placeholder parameter for the 2 successive steps. **Describe the solution you'd like** Add a field called prerequisites in the pipelinecommandjob yaml so that i can specify which steps need to complete before starting the current step. **Describe alternatives you've considered** **Additional context**
Author: newhardwarefound
Assignees: -
Labels: `Service Attention`, `Machine Learning`, `customer-reported`
Milestone: -
Uranium2 commented 11 months ago

It is indeed very frustrating. For me it's the only blocking point to migrate my scripts to V2.

tdegamboda commented 11 months ago

Are there any updates on this?

Uranium2 commented 11 months ago

@tdegamboda @newhardwarefound I made a tool on my own to get something similar, to create a worflow of steps (command/node in V2 namming) without worring about the Inputs/Outputs. It will run your steps in the order you gave in a graph.

https://stackoverflow.com/a/77354028/5462743