Azure AppServices fails pulling an image while it is confirmed to be present in ACR by CLI

[blacelle]

This is a Bug Report

I succeeded pushing a docker img in ACR with spotify maven plugin. I confirms both in Azure portal and through 'docker pull mitrust.azurecr.io/devlocalnodejs-mitrust:devlocalnodejs-20180827193841815-5c8e9c3' the image does exist. However, the pull with Azure App Services fails, while the same configuration used to work since weeks:

2018-08-27 19:58:27.129 INFO  - Issuing docker pull mitrust.azurecr.io/devlocalnodejs-mitrust:devlocalnodejs-20180827193841815-5c8e9c3
2018-08-27 19:58:27.234 ERROR - docker pull returned STDERR>> Error response from daemon: Get https://mitrust.azurecr.io/v2/devlocalnodejs-mitrust/manifests/devlocalnodejs-20180827193841815-5c8e9c3: unauthorized: authentication required

2018-08-27 19:58:27.239 INFO  - Starting container for site
2018-08-27 19:58:27.241 INFO  - docker run -d -p 63699:8080 --name devlocalnodejs-mitrust_0 -e WEBSITES_PORT=8080 -e DOCKER_CUSTOM_IMAGE_NAME=mitrust.azurecr.io/devlocalnodejs-mitrust:devlocalnodejs-20180827193841815-5c8e9c3 -e WEBSITE_SITE_NAME=devlocalnodejs-mitrust -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_INSTANCE_ID=6b1444909d8972005433e54176839a014641e94e2f5e9fd6965ca0a1b499d8a6 -e HTTP_LOGGING_ENABLED=1 mitrust.azurecr.io/devlocalnodejs-mitrust:devlocalnodejs-20180827193841815-5c8e9c3  

2018-08-27 19:58:27.285 ERROR - Container create failed for devlocalnodejs-mitrust_0 with System.AggregateException, One or more errors occurred.
InnerException: Docker.DotNet.DockerContainerNotFoundException, Docker API responded with status code=NotFound, response={"message":"No such image: mitrust.azurecr.io/devlocalnodejs-mitrust:devlocalnodejs-20180827193841815-5c8e9c3"}

[sajayantony]

@mmacy @SteveLasker - this should be an App Service Issue so not sure how to route this to them.

[yiliaomsft]

did you try to restart the site and check if the pull is able to succeed?

[blacelle]

Hello. No, even after restarting the site, the image is not pulled. Yesterday, I succeeded pulling a new image from another repository/AppService, but this morning, even this second repository/AppService fails pulling images.

I also tried clearing the ACR repository, without success

[yiliaomsft]

Hi, do you mind to check if you enabled Admin user for your Azure Container Registry? which is required for App Service.

[blacelle]

Hi. It is activated. I guess this is not a configuration issue as this issue ends being solved by itself after a few hours/days, before appearing again.

[yiliaomsft]

Ok thanks for confirming it. Do you mind to share the site name so we can take a look?

[blacelle]

Thanks for having a deeper look. I'n getting anxious as all my environments are getting cluttered with this:

dev-mitrust devlocalnodejs-mitrust uat-mitrust

[sajayantony]

@yiliaomsft - do you have have a place where we can move this issue to or do you want to keep this communication here. We are fine either way.

[yiliaomsft]

@blacelle is this still an issue? Please contact me directly if you still see the issue and we can create a support case to track it.

[elsesiy]

I'm facing the very same issue with a similar setup. I tried updating the admin credentials, restarts, remove the credentials and re-select the image to be used so it establishes the connection automatically again and nothing works. The only thing working for me is deleting the whole thing and start from scratch. We actually face this issue now also while auto-scaling our environments where based on load additional instances are supposed to be spun up but it fails due to the described error.

2018-11-09 23:16:12.886 INFO  - Issuing docker pull ourregistryhere.azurecr.io/tag:version
2018-11-09 23:16:13.154 ERROR - docker pull returned STDERR>> Error response from daemon: Get https://ourregistryhere.azurecr.io/v2/project/manifests/version: unauthorized: authentication required

2018-11-09 23:16:13.178 INFO  - Starting container for site
2018-11-09 23:16:13.180 INFO  - docker run -d -p 30811:80 --name xyz -e WEBSITES_ENABLE_APP_SERVICE_STORAGE=false -e WEBSITE_SITE_NAME=internal-api-amap-int -e WEBSITE_AUTH_ENABLED=False -e PORT=80 -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_INSTANCE_ID=679e7adbe590fdb21251d27e0e66375f03cfaeb147447885bccba4565fb6192f -e HTTP_LOGGING_ENABLED=1 ourregistryhere.azurecr.io/tag:version

2018-11-09 23:16:13.205 ERROR - Container create failed for xyz with System.AggregateException, One or more errors occurred.
InnerException: Docker.DotNet.DockerContainerNotFoundException, Docker API responded with status code=NotFound, response={"message":"No such image: ourregistryhere.azurecr.io/tag:version"}

I redacted the logs due to confidentiality issues but there's nothing really in there helping to debug the issue. I'm happy to jump on a call with the support if needed, but we need to get this resolved quickly.

Thanks!

[SteveLasker]

@yiliaomsft Can you help out here?

[yiliaomsft]

elsesiy - please send me your site name by e-mail and we will take a look.

[elsesiy]

@yiliaomsft I created a support ticket which has been bumped up to Severity A. I'm talking to the support engineer Christopher Maldonado who should be coordinating with you guys already. Please let me know if you need any further info other then the diagnostics info I already provided via e-mail.

[methtt]

hi @elsesiy , wondering if you've had any follow up or updates regarding this issue?

[elsesiy]

I was able to resolve the issue. The error message is completely misleading as the error is not related to ACR authentication at all but rather how Azure executes containers in userns.

Check out this guide from CircleCI on how to resolve this issue.

I wish Azure would propagate issues with docker pull/run back to the client and not just some useless catch-all & unrelated message. If I had known this I could've saved two weeks going back and forth with the support.

[deus42]

Faced the same issue, my problem was that I didn't publish 'latest' tag to the ACR. That fixed the error. The unauthorized error message is irrelevant.

[sajayantony]

Closing this since it is handled by App Service.

[ashtonian]

wow, thank you - ran into a similar issue. Was getting not authorized from app services when it was trying to pull the image from acr. The docker image was being built with azure devops docker build and push steps. Agent was from a windows based pool and this apparently causes problems when using azure app services with a container on a linux host. Documentation is cool.

[AdalynRockwood]

@Ashtonian - How did you fix that?

[its-saurabhjain]

Same issue, the ACR is having my docker image and i am able to deploy that image in to Azure Container Instance, however deploying the same image App Service for Container using CLI getting unauthorized error.

[ashtonian]

@AdalynRockwood use a *nix (Hosted Ubuntu 1604) based pool agent when deploying to nix and windows when windows. If I remember correctly its a pretty common file permissions issue with docker. You should be able to get a replicate it by deploying a windows image on nix by hand to get the actual docker error that app services is wrapping with the auth error.

[zapap]

Having same issue today. Admin permission enabled in ACR. My Azure DevOps Build pipeline is using Ubuntu 16.04 agent, builds and pushed the docker image successfully to ACR. My Azure DevOps Release pipeline is reporting success too. My App Service is configured for docker on Linux. Single container using ACR and Continuous Deployment is Off. Looking at my app service's container settings, I can see the image tag has been updated correctly by the release, but in the logs I still get this error:

2019-05-26 01:53:30.825 ERROR - Pulling docker image ****containers/webapi***:20190526.08.285 failed:
2019-05-26 01:53:30.825 INFO  - Pulling image from Docker hub: ****containers/webapi***:20190526.08.285
2019-05-26 01:53:30.892 ERROR - DockerApiException: Docker API responded with status code=NotFound, response={"message":"pull access denied for ****containers/webapi***, repository does not exist or may require 'docker login'"}

If I now enable Continuous Deployment, the image is pulled successfully and my application works as expected.

Any ideas what I can try next?

[zapap]

Solved: In Azure DevOps Release pipeline, I had to set the Registry or Namespace on the "Deployment Process" level to the fully qualified name: containers.azurecr.io. I initially only had it as containers.

It also appears you need to create a new release after updating the setting, as the updated value is not applied to existing releases (for re-deployments)

In my case adding an app registration in AAD and a service principal on the subscription with the reader role didn't solve it. I removed all this and the deployment still works - so this had no affect.

[sieunhantanbao]

Solved: In Azure DevOps Release pipeline, I had to set the Registry or Namespace on the "Deployment Process" level to the fully qualified name: containers.azurecr.io. I initially only had it as containers.

It also appears you need to create a new release after updating the setting, as the updated value is not applied to existing releases (for re-deployments)

In my case adding an app registration in AAD and a service principal on the subscription with the reader role didn't solve it. I removed all this and the deployment still works - so this had no affect.

Thank you very much. I had the same issue. The problem was solved with your solution.

[m-sterspace]

I encountered this rather non descript error myself today as well. For me it was caused because when I flipped through creating the app service, I had selected quick start which had defaulted it's settings to docker hub instead of ACR. Changing it to ACR while leaving continuous deployment off let my pipelines start working properly.

[rainabba]

Apparently, sufficiently large images will cause this as well, even when logs indicate "Pull complete"

2020-02-15 21:56:27.346 INFO  - Pulling image: myOrg/myRepo::latest
2020-02-15 21:56:28.065 INFO  - 4.0.27 Pulling from myOrg/myRepo
2020-02-15 21:56:28.068 INFO  - 54d7e8ac135a Already exists
2020-02-15 21:56:28.076 INFO  - d6341e31912f Already exists
.... a while later and lots of downloads/extractions
2020-02-15 22:00:19.041 INFO  - 6aba6bd666df Pull complete
2020-02-15 22:00:19.043 INFO  -  Digest: sha256:d99913563611f03cdac48edaa398206fc8ba79cc363f4a529be0da1b9732ce356
2020-02-15 22:00:19.044 INFO  -  Status: Downloaded newer image for myOrg/myRepo
2020-02-15 22:00:19.056 INFO  - Pull Image failed, Time taken: 3 Minutes and 51 Seconds
2020-02-15 22:00:19.057 ERROR - Pulling docker image myOrg/myRepo::latest failed:
2020-02-15 22:00:19.058 INFO  - Pulling image from Docker hub: myOrg/myRepo::latest
2020-02-15 22:00:19.582 ERROR - DockerApiException: Docker API responded with status code=NotFound, response={"message":"pull access denied for myOrg/myRepo, repository does not exist or may require 'docker login': denied: requested access to the resource is denied"}

2020-02-15 22:00:19.585 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2020-02-15 22:00:19.615 INFO  - Stoping site mysite because it failed during startup.

[m0m01hl]

Facing same issue today. Admin permission enabled in ACR. My Azure DevOps Build pipeline is using Linux Build Agent, builds and pushed the docker image successfully to ACR. My Azure DevOps Release pipeline is reporting success too. My App Service is configured for docker on Linux. Single container using ACR and Continuous Deployment is Off. Looking at my app service's container settings, I can see the image tag has been updated correctly by the release, but in the logs I still get this error: 2020-02-27 20:26:50.967 INFO - Pulling image: gbsfinapdev.azurecr.io/ap-fds-otl-claimrefid:113127 2020-02-27 20:26:51.098 ERROR - DockerApiException: Docker API responded with status code=NotFound, response={"message":"manifest for gbsfinapdev.azurecr.io/ap-fds-otl-claimrefid:113127 not found: manifest unknown: manifest unknown"}

2020-02-27 20:26:51.099 ERROR - Pulling docker image gbsfinapdev.azurecr.io/ap-fds-otl-claimrefid:113127 failed: 2020-02-27 20:26:51.100 INFO - Pulling image from Docker hub: gbsfinapdev.azurecr.io/ap-fds-otl-claimrefid:113127 2020-02-27 20:26:51.124 ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://gbsfinapdev.azurecr.io/v2/ap-fds-otl-claimrefid/manifests/113127: unauthorized: authentication required"}

2020-02-27 20:26:51.126 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository) 2020-02-27 20:26:51.135 INFO - Stoping site apfdsotl-crfd because it failed during startup. 2020-02-27 20:31:50.366 INFO - Pulling image: gbsfinapdev.azurecr.io/ap-fds-otl-claimrefid:113127 2020-02-27 20:31:50.515 ERROR - DockerApiException: Docker API responded with status code=NotFound, response={"message":"manifest for gbsfinapdev.azurecr.io/ap-fds-otl-claimrefid:113127 not found: manifest unknown: manifest unknown"}

2020-02-27 20:31:50.516 ERROR - Pulling docker image gbsfinapdev.azurecr.io/ap-fds-otl-claimrefid:113127 failed: 2020-02-27 20:31:50.516 INFO - Pulling image from Docker hub: gbsfinapdev.azurecr.io/ap-fds-otl-claimrefid:113127 2020-02-27 20:31:50.540 ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://gbsfinapdev.azurecr.io/v2/ap-fds-otl-claimrefid/manifests/113127: unauthorized: authentication required"}

2020-02-27 20:31:50.542 ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository) 2020-02-27 20:31:50.543 INFO - Stoping site apfdsotl-crfd because it failed during startup. can someone please me with this.

[katatohuk]

have the same issue today

[seiffs]

I have the same issue

[seiffs]

I have the same issue

Ok, I finally resolved my issue. I had a typo in the repository name! Easy test is to run: docker pull "the-url-from-your-error-message" Make sure to be logged to the repository in the console first: az acr login --name yourcontainerregistry

[SteveLasker]

Hey folks, Apologies for the challenges with authenticating with ACR for App services. We've been enabling self service diagnostics to help troubleshoot typos, changed passwords and other challenges.

• If you're having auth problems, please take a look a the ACR Diagnostics & Logs features: https://aka.ms/acr/audit-logs. If you don't see an entry, check the registry name, or general connectivity.
• When you can get onto the box with challenges, you can also use the az acr health-check api: https://aka.ms/acr/health-check for connectivity verification.

Hope this helps Steve

[jishnukoottala]

• I also got the same issue Here is how I got it resolved
• Go to Azure Container Registry, enable Admin Access
• Goto IAM Control, add a contributor role to App Service( if app service is not reflecting, go to your app service and go to identity section on the left, and click on system assigned - change status to On)
• Restart your App Service, changes will take few minutes to propogate

[prajith234]

@jishnukoottala This solution worked for me , Perfect ..!

[harishpo]

Solved: In Azure DevOps Release pipeline, I had to set the Registry or Namespace on the "Deployment Process" level to the fully qualified name: containers.azurecr.io. I initially only had it as containers.

It also appears you need to create a new release after updating the setting, as the updated value is not applied to existing releases (for re-deployments)

In my case adding an app registration in AAD and a service principal on the subscription with the reader role didn't solve it. I removed all this and the deployment still works - so this had no affect.

Thank you , it worked for me.

[ghost]

• I also got the same issue Here is how I got it resolved
• Go to Azure Container Registry, enable Admin Access
• Goto IAM Control, add a contributor role to App Service( if app service is not reflecting, go to your app service and go to identity section on the left, and click on system assigned - change status to On)
• Restart your App Service, changes will take few minutes to propogate

Your solution worked for me too, thanks.

[SteveLasker]

Once you're configured, you should be able to disable the admin account. This configuration limitation is something the App Service Team is working to resolve as the admin account has full push/pull permissions and only one identity. If you need a more holistic token solution, we now have repo-scoped token support, with RBAC. https://aka.ms/acr/repo-permissions However, for Azure Services, managed identities are the most secure solution as you don't have to manage key rotation. Apologies to those that have struggled and had to find this doc. Steve

[AswinPM]

• I also got the same issue Here is how I got it resolved
• Go to Azure Container Registry, enable Admin Access
• Goto IAM Control, add a contributor role to App Service( if app service is not reflecting, go to your app service and go to identity section on the left, and click on system assigned - change status to On)
• Restart your App Service, changes will take few minutes to propogate

This worked when I provided ACR pull role instead of the contributor role.

[mossherder]

I had this issue and solved it differently than the suggested answers above. For us, we set the ACR credentials in the App Service Application Settings under these three variables.

[lmammino]

I followed all the commands in this beautiful Azure tutorial and ended up having the same error:

2023-02-24T10:10:12.469Z INFO - Logging is not enabled for this container.
Please use https://aka.ms/linux-diagnostics to enable logging to see container logs here.
2023-02-24T10:12:49.835Z ERROR - DockerApiException: Docker API responded with status code=InternalServerError, response={"message":"Get https://crazyunicorn.azurecr.io/v2/appsvc-tutorial-custom-image/manifests/latest: unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information."}

2023-02-24T10:12:49.836Z WARN - Image pull failed. Defaulting to local copy if present.
2023-02-24T10:12:49.843Z ERROR - Image pull failed: Verify docker image configuration and credentials (if using private repository)
2023-02-24T10:12:55.767Z INFO - Stopping site crazyunicornapp because it failed during startup.
0dd1be2_msiProxy -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=crazyunicornapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=[redacted] -e WEBSITE_USE_DIAGNOSTIC_SERVER=False mcr.microsoft.com/appsvc/msitokenservice:stage4

I have the applications settings set up (as per the tutorial):

After spending a couple of days trying to troubleshoot this (reading several very different forums with very different solutions) I am starting to feel the itch to go back to Fargate...

Any final suggestion on how I could troubleshoot this problem?

Was anyone able to follow the tutorial successfully?