Job naming issues with AML CLI training pipeline

[luhgit]

Why?

Currently when we deploy training pipeline using AML CLI we run into the problem that job names could not be validated.

How?

As we can see in the screenshot above that hyphens ain't allowed in the naming conventions for the jobs. So we should consider repacing hyphens with underscores in pipeline.yml file.

I could also create a PR if you consider this as a valid way of solving this issue.

Anything else?

[cynthiajiangatl]

I replaced all the hyphens with underscores, it didn't fix the error. Please let me know how to get the pipeline to run using AML CLI2

[lostmygithubaccount]

@cynthiajiangatl what is the error you are seeing? If you replaced the hyphens with underscores, you shouldn't be seeing the same error

[cynthiajiangatl]

I still see the same error, not sure why it was not using the updated code. When I tried to use a particular commit to run the devops pipeline, it ran into another issue, just not checking out the latest code for some reason.

[lokijota]

Hi @cynthiajiangatl . For some reason ADO didn't pick up the changes with me either, I had to recreate the pipeline in ADO, but then it worked. Just make sure you replace all the references in the yml file, in the job stage names and references.

[cynthiajiangatl]

@lokijota , thank you for the advice. I deleted the pipeline and created a new one, it did take the updated code and ran the pipeline successfully. Could someone from Engineering take a look at the Github code checkout issue? Thanks.

[buswrecker]

I have the same error also. Are '-' dashes allowed? the MLOps-v2 uses dashes. https://github.com/Azure/mlops-v2/blob/main/data-science-regression/pipeline.yml. The official AML Examples - https://github.com/Azure/azureml-examples/blob/main/cli/jobs/pipelines/nyc-taxi/pipeline.yml YML uses underscores "_". What's the best practice?

[msdpalam]

I had the same issue; I renamed the job names under data-science-regression/pipeline.yml, and then made sure the other step jobs reflect the changes. This resolved my issue. Thought I would share: jobs: prep_job: type: command component: file:./components/prep/prep.yml inputs: raw_data: ${{parent.inputs.pipeline_job_input}} outputs: transformed_data: ${{parent.outputs.pipeline_job_transformed_data}}

train_job: type: command component: file:./components/train/train.yml inputs: training_data: ${{parent.jobs.prep_job.outputs.transformed_data}} outputs: model_output: ${{parent.outputs.pipeline_job_trained_model}} .... ....

[andreipurdila]

Regarding the ADO pipeline not picking up the latest changes, this is due to the presence of the Cache@2 task in the install-az-cli.yml template. Remove that step (or properly configure it) and the ADO pipeline will pick up the latest changes.

[cindyweng]

removed Cache@2 task in the cli-v2 template as Andrei specified, code changes elsewhere now will be reflected when you rerun the same pipeline