Azure / MachineLearningNotebooks

Python notebooks with ML and deep learning examples with Azure Machine Learning Python SDK | Microsoft
https://docs.microsoft.com/azure/machine-learning/service/
MIT License
4.07k stars 2.52k forks source link

Parallel execution unable to pull Docker image #1643

Closed Katzmann1983 closed 2 years ago

Katzmann1983 commented 2 years ago

I am running the example file-dataset-partition-per-folder.ipynb and tried to change the environment to a Docker image used successfully before within ml.azure.com. The custom environement has a parent image, including username and password. When tring to use it in the example notebook the following errors are thrown:

Streaming azureml-logs/55_azureml-execution-tvmps_272fd8ee3ed11e9518670a619abc06e29ea9746436a19366961d6b492c855998_p.txt

2021-11-24T18:43:48Z Running following command: /bin/bash -c sudo blobfuse /mnt/batch/tasks/shared/LS_root/jobs/mldevdvlgendrdwe02-ml/azureml/633dd4ba-a75e-404e-80b0-7f2023e84d8b/mounts/workspaceblobstore --tmp-path=/mnt/batch/tasks/shared/LS_root/jobs/mldevdvlgendrdwe02-ml/azureml/633dd4ba-a75e-404e-80b0-7f2023e84d8b/caches/workspaceblobstore --file-cache-timeout-in-seconds=1000000 --cache-size-mb=11455 -o nonempty -o allow_other --config-file=/mnt/batch/tasks/shared/LS_root/jobs/mldevdvlgendrdwe02-ml/azureml/633dd4ba-a75e-404e-80b0-7f2023e84d8b/configs/workspaceblobstore.cfg --log-level=LOG_WARNING 2021-11-24T18:43:48Z Successfully mounted a/an Blobfuse File System at /mnt/batch/tasks/shared/LS_root/jobs/mldevdvlgendrdwe02-ml/azureml/633dd4ba-a75e-404e-80b0-7f2023e84d8b/mounts/workspaceblobstore 2021-11-24T18:43:49Z The vmsize standard_ds2_v2 is not a GPU VM, skipping get GPU count by running nvidia-smi command. 2021-11-24T18:43:49Z Starting output-watcher... 2021-11-24T18:43:49Z IsDedicatedCompute == False, starting polling for Low-Pri Preemption 2021-11-24T18:48:49Z The vmsize standard_ds2_v2 is not a GPU VM, skipping running nvidia-smi command. 2021-11-24T18:49:08Z Running Docker Command attempt 1 failed with err: exit status 1,Error response from daemon: Head "https://acrdevdvlstr01.azurecr.io/v2/mlops/manifests/v01": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.

2021-11-24T18:49:13Z Retry Docker Command... 2021-11-24T18:49:13Z Running Docker Command attempt 2 failed with err: exit status 1,Error response from daemon: Head "https://acrdevdvlstr01.azurecr.io/v2/mlops/manifests/v01": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.

2021-11-24T18:49:22Z Retry Docker Command... 2021-11-24T18:49:22Z Running Docker Command attempt 3 failed with err: exit status 1,Error response from daemon: Head "https://acrdevdvlstr01.azurecr.io/v2/mlops/manifests/v01": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.

2021-11-24T18:49:38Z Retry Docker Command... 2021-11-24T18:49:38Z Running Docker Command attempt 4 failed with err: exit status 1,Error response from daemon: Head "https://acrdevdvlstr01.azurecr.io/v2/mlops/manifests/v01": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.

2021-11-24T18:50:11Z Retry Docker Command... 2021-11-24T18:50:11Z Running Docker Command attempt 5 failed with err: exit status 1,Error response from daemon: Head "https://acrdevdvlstr01.azurecr.io/v2/mlops/manifests/v01": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.

2021-11-24T18:50:16Z Removing container 633dd4ba-a75e-404e-80b0-7f2023e84d8b exited with 0, Error: No such container: 633dd4ba-a75e-404e-80b0-7f2023e84d8b

Katzmann1983 commented 2 years ago

It was in fact an authorization issue outside of ml.azure.com. Fixed it