Install AZ ML dependencies step is failing during deploy-model-training-pipeline.yml execution

shishirdash24 commented 4 months ago

Describe the bug or the issue that you are facing

deploy-model-training-pipeline.yml was working totally fine till last Friday & suddenly we have started facing issue with it from Monday. Install AZ ML dependencies is failing with the below error: /usr/bin/bash /home/vsts/work/_temp/azureclitaskscript1721726661839.sh Collecting pip Downloading pip-24.1.2-py3-none-any.whl (1.8 MB) ERROR: Could not find a version that satisfies the requirement install (from versions: none) ERROR: No matching distribution found for install

Please see the attached image for more information.

Steps/Code to Reproduce

Clone & configure mlops-v2, mlops-project-template & mlops-template
Create a new repo for a project & initialize the project repo using (https://github.com/Azure/mlops-v2/blob/main/documentation/deployguides/deployguide_ado.md#create-and-configure-a-new-ml-project-repo). Make sure to use MLOps interface as ' python-sdk-v2' & Infrastructure provider as 'Bicep'.
Execute bicep-ado-deploy-infra.yml pipeline
Execute deploy-model-training-pipeline.yml pipeline (this is where you can see the error)

Expected Output

There shall not be any error in the script

Versions

main branch of the repo in DevOps. I am using Azure ML SDK V2

Which platform are you using for deploying your infrastrucutre?

Azure DevOps (ADO)

If you mentioned Others, please mention which platformm are you using?

No response

What are you using for deploying your infrastrucutre?

Bicep

Are you using Azure ML CLI v2 or Azure ML Python SDK v2

Azure ML Python SDK v2

Describe the example that you are trying to run?

Execute deploy-model-training-pipeline.yml pipeline

shishirdash24 commented 3 months ago

install-az-cli.yml in the template contains [python -m pip install -U --force-reinstall pip pip install ==]. Seems like this is causing the failure. I fixed it by modifying the commands to [ python -m pip install -U ==]. But now the next step in the execution which runs install-aml-cli.yml is failing with error: AttributeError: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'

[error]Script failed with error: WARNING: Auto upgrade failed. name 'exit_code' is not defined

Please help getting a solution.

setuc commented 3 months ago

It seems there is an issue with the pip upgrade. I will investigate this next week. Possibly an older version is causing an issue together with the cryptography module.

murali513 commented 3 months ago

We are also getting same kind of error from July 17th :

Please let me know if any body any solution for this issue.

shishirdash24 commented 3 months ago

Microsoft is yet to provide any solution to this, but I found out some workaround though.

If you check the logs, "python -m pip install -U --force-reinstall pip" in mlops-templates -> templates -> aml-cli-v2 -> install-az-cli.yml is the culprit here.

Remove "python -m pip install -U --force-reinstall pip" from each line. python -m pip install -U --force-reinstall pip python -m pip install -U azure-cli==2.35 python -m pip install -U azure-ai-ml==1.1.0 python -m pip install -U azure-common==1.1.28 python -m pip install -U azure-core==1.26.1 python -m pip install -U azure-identity==1.10.0 python -m pip install -U azure-mgmt-core==1.3.0 python -m pip install -U azure-storage-blob==12.14.1 python -m pip install -U azure-storage-file-datalake==12.9.1 python -m pip install -U azure-storage-file-share==12.7.0

It may fail in the next step again(which is Install AML CLI V2) because of a missing package. Add the below command to install-az-cli.yml to fix the issue: python -m pip install --upgrade pyOpenSSL cryptography requests

jason-theodorakopoulos commented 4 weeks ago

The above workaround worked fined. @setuc can you reflect this change?

Azure / mlops-v2