search

solliancenet / tech-immersion-data-ai

123 stars 143 forks source link

ai-exp6 - knack.util.CLIError: Could not retrieve credential from local cache for service principal ***. Run `az login` for this service principal. #48

Closed krisbock closed 2 years ago

krisbock commented 2 years ago

Task 2: Run the Build Pipeline fails in 'Get or create AML Compute Target' step with:

2022-01-06T05:53:56.435590] Logging experiment finalizing status in history service. Starting the daemon thread to refresh tokens in background for process with pid = 3694 Cleaning up all outstanding Run operations, waiting 300.0 seconds 1 items cleaning up... Cleanup took 0.1008906364440918 seconds Traceback (most recent call last): File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azureml/core/authentication.py", line 2283, in _get_arm_token_with_refresh access_token = profile_object.get_raw_token(resource=resource)[0][1] File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 383, in get_raw_token credential = self._create_credential(account, tenant) File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 597, in _create_credential return identity.get_service_principal_credential(username_or_sp_id) File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/auth/identity.py", line 236, in get_service_principal_credential entry = self._service_principal_store.load_entry(client_id, self.tenant_id) File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/auth/identity.py", line 320, in load_entry .format(sp_id)) knack.util.CLIError: Could not retrieve credential from local cache for service principal ***. Run az login for this service principal.

Error from AML driver log: n create_aml_cluster.py Azure ML SDK version: 1.37.0 Argument 1: gpucluster Argument 2: /home/vsts/work/1/s creating AzureCliAuthentication... done creating AzureCliAuthentication! get workspace...

[2022-01-06T05:53:56.425879] The experiment failed. Finalizing run... [2022-01-06T05:53:56.425904] Start FinalizingInRunHistory [2022-01-06T05:53:56.435590] Logging experiment finalizing status in history service. Starting the daemon thread to refresh tokens in background for process with pid = 3694 Cleaning up all outstanding Run operations, waiting 300.0 seconds 1 items cleaning up... Cleanup took 0.1008906364440918 seconds Traceback (most recent call last): File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azureml/core/authentication.py", line 2283, in _get_arm_token_with_refresh access_token = profile_object.get_raw_token(resource=resource)[0][1] File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 383, in get_raw_token credential = self._create_credential(account, tenant) File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/_profile.py", line 597, in _create_credential return identity.get_service_principal_credential(username_or_sp_id) File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/auth/identity.py", line 236, in get_service_principal_credential entry = self._service_principal_store.load_entry(client_id, self.tenant_id) File "/home/vsts/.azureml/envs/azureml_b54a7a69e83d04a100a32a3a44b87b62/lib/python3.6/site-packages/azure/cli/core/auth/identity.py", line 320, in load_entry .format(sp_id)) knack.util.CLIError: Could not retrieve credential from local cache for service principal XXXXXXXXXXXXX. Run az login for this service principal.

Looks like the error is caused by breaking changing in azcli with tokens: https://github.com/Azure/cli/issues/56

krisbock commented 2 years ago

Looks like a number of PRs have been issued for the source repo which needs the CLI version updated: https://github.com/solliancenet/mcw-mlops-starter/pull/3

i.e. updating environment_setup/install_requirements.sh to python --version pip install azure-cli==2.32.0 pip install azureml-sdk==1.37.0 pip install azureml-dataprep[pandas]