`az container create` throwing `ERROR: (RegistryErrorResponse) An error response is received from the docker registry 'index.docker.io'. Please retry later.`

[casions]

Describe the bug

When trying to create a container using the Azure CLI Container Create command, we get a RegistryErrorResponse error. This is trying to deploy an image from an Azure private registry.

This was working yesterday with the same version as today: 2.61.0

I can supply more of my own configuration on request if necessary.

Related command

az container create --resource-group $RESOURCE_GROUP --file $ACI_FILE

Errors

ERROR: (RegistryErrorResponse) An error response is received from the docker registry 'index.docker.io'. Please retry later.

Issue script & Debug output

Unable to provide this at this time.

Expected behavior

The container should be created.

Environment Summary

azure-cli 2.61.0

core 2.61.0 telemetry 1.1.0

Extensions: azure-devops 1.0.1

Dependencies: msal 1.28.0 azure-mgmt-resource 23.1.1

Python location '/opt/az/bin/python3' Extensions directory '/opt/az/azcliextensions'

Python (Linux) 3.11.8 (main, May 16 2024, 03:47:28) [GCC 11.4.0]

Additional context

Another person having the exact same issue since today with a public registry.

- task: AzureCLI@2
  name: CreateContainer
  displayName: Create container
  inputs:
    azureSubscription: ${{ parameters.subscription }}
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      az container create \
        --resource-group '${{ parameters.resourceGroup }}' \
        --name '${{ parameters.name }}' \
        --image 'docker.io/mockoon/cli:latest' \
        --ports ${{ parameters.port }} \
        --location '${{ parameters.location }}' \
        --dns-name-label '${{ parameters.name }}' \
        --command-line "mockoon-cli start --data '$(publicUrl)' --port ${{ parameters.port }}"`

link

[yonzhan]

Thank you for opening this issue, we will look into it.

[danspam]

We are also experiencing this issue for at least the last 18 hours. When running the az container create command either locally in the command line or during an azure devops pipeline. The issue appears to be intermittent, failing more times that it is successful. It is not the same image all the time that fails.

Example command:

az container create --resource-group my-resource --name container-mysql-xxx --image mysql:8.0 --command-line "/usr/local/bin/docker-entrypoint.sh --lower-case-table-names=1 --collation-server=utf8mb4_unicode_ci" --environment-variables MYSQL_ROOT_PASSWORD=thepassword --location australiaeast --os-type Linux --restart-policy Never --cpu 1 --memory 1 --ports 3306 --dns-name-label container-mysql-xxx --no-wait

[Taylor-S]

Hey guys,

Thanks for looking into this. We have been experiencing this issue on and off for the past couple of weeks. But as of the past 24 hours it's been consistently failing. It's preventing our deployment process from going out. Here's what our pipeline log looks like.

Your CLI is up-to-date.
Setting AZURE_CONFIG_DIR env variable to: /home/vsts/work/_temp/.azclitask
Setting active cloud to: AzureCloud
/usr/bin/az cloud set -n AzureCloud
/usr/bin/az login --service-principal -u *** --password=*** --tenant df3ef93a-489d-4317-8cee-047a66225cc9 --allow-no-subscriptions
[
  {
    "cloudName": "AzureCloud",
    "homeTenantId": "df3ef93a-489d-4317-8cee-047a66225cc9",
    "id": "df3ef93a-489d-4317-8cee-047a66225cc9",
    "isDefault": true,
    "managedByTenants": [],
    "name": "COMPANY AZURE",
    "state": "Enabled",
    "tenantId": "df3ef93a-489d-4317-8cee-047a66225cc9",
    "user": {
      "name": "***",
      "type": "servicePrincipal"
    }
  }
]
/usr/bin/az account set --subscription df3ef93a-489d-4317-8cee-047a66225cc9
/usr/bin/bash /home/vsts/work/_temp/azureclitaskscript1719962335202.sh
ERROR: (RegistryErrorResponse) An error response is received from the docker registry 'index.docker.io'. Please retry later.
Code: RegistryErrorResponse
Message: An error response is received from the docker registry 'index.docker.io'. Please retry later.
##[error]Script failed with exit code: 1
/usr/bin/az account clear
Finishing: Azure CLI 

[abarqawi]

@casions @Taylor-S @danspam This related to rate limiting Azure & Docker hub , the issue described here

https://medium.com/@alaa.barqawi/docker-rate-limit-with-azure-container-instance-and-aks-4449cede66dd

[casions]

I think I can now confirm this @abarqawi. Initially I was sceptical as we were using an image from ACR, not from the public registry. However, we have now determined that we also pull an image from the public registry at the same time and this was causing the failure due to the rate limiting as mentioned. The workaround is to copy the public image to the Azure Container Registry and pull it from there. I still don't think that we are hitting the free limit though and think that this could be caused by some sort of shared resource.

[danspam]

Thanks @abarqawi. Time to change to azure container registry...

[Taylor-S]

Great! Thanks for that @abarqawi

[avalanche-tm]

I don't understand, it says unauthenticated users can pull up to 100 images per 6-hour period. I pull 2-3 images and get this error.

[Prudhvi469]

Any update on this issue ?

[casions]

@avalanche-tm 100 pulls per 6 hours per IP address. Who else is using the IP address? https://docs.docker.com/docker-hub/download-rate-limit/

[magliok-wwt]

Can anyone assist more on this??

I don't want to spin up an ACR - I just want to use docker.io authenticated using the az container create command

I would prefer not to have to do a weird hybrid local docker pulling / pushing to an ACR - again we're back to something that should be able to be handled all cloud-native.

[abarqawi]

@magliok-wwt you can try to create ACI az follows https://learn.microsoft.com/en-us/cli/azure/container?view=azure-cli-latest#az-container-create you need to set those parameters in order to use docker credential az container create --resource-group

                [--registry-login-server]
                [--registry-password]
                [--registry-username]

Something like: az container create --resource-group FRONT-SELECT-NA2 --registry-login-server jfrogtraining-docker-dev.jfrog.io --registry-username svc-faselect --registry-password "..." --file ads-azure.yaml

[magliok-wwt]

Yeah unfortunately that requires you stand up an entire ACR, have docker locally running and manage images pull / push.

I wanted to be able to pull an image - directly from docker.io registry - just authenticated.

I just haven't seen anyone use those switches for anything other than an ACR, gotta work for docker though too right?

[abarqawi]

Yeah unfortunately that requires you stand up an entire ACR, have docker locally running and manage images pull / push.

I wanted to be able to pull an image - directly from docker.io registry - just authenticated.

I just haven't seen anyone use those switches for anything other than an ACR, gotta work for docker though too right?

This will create ACI from authenticated dockerhub without using ACR , az container create you need to create account for private registry in dockerhub first https://docs.docker.com/subscription/core-subscription/details/

[arthur-twingate]

@magilok-wwt It's two weeks later but I just spent some time wrestling with this bear and can tell you you'll need to add: --registry-username [dockerhub username] --registry-password [dockerhub password] --registry-login-server index.docker.io to your az container create command and it should work fine.

Seems like unauthenticated requests between Azure and DockerHub reflect as the "same" account/ip/whatever and get throttled but the minute you drop in credentials, it works as expected without having to create a new ACR just to pull a public image.

[magliok-wwt]

So I ended up figuring it out - as my want was to not have to utilize a local docker daemon which was even more painful....

But for anyone else trying to simply pull a docker image, into an Azure container, without having to pull local onto their machine, tag, and push back up... keeping it all within Azure on nice fast connections.

1. You need to have an ACR that allows an access token to be generated through CLI - 'admin mode' needs to be enabled for this to fully work all the way through so that the az container create command can auth and hook to your own ACR. So first go create an ACR and ensure admin mode is enabled.

2. Then to pull/push an image: open the Azure Cloud Shell (using powershell below not bash fwiw)

3. Run the following

   • The special sauce is the docker login command that allows you to utilize this import command without a daemon running. This also logs you into your ACR in Azure automatically, so that the import command seamlessly works.
   • The username and password in the import command are utilized so that you get around docker throttling limits utilizing your own account ( using a docker PAT on your account )
   • Even SSO users can ( and have to ) utilize this method
   • AppSmith image is being used as an example, as they have multiple images we are placing in appsmith/

$acr = "nameOFyourACRhere"  
$acrToken = $(az acr login --name $acr --expose-token --query "accessToken" -o tsv)
$acrLS = $(az acr show --name $acr --query loginServer --output tsv)
docker login $acrLS -u 00000000-0000-0000-0000-000000000000 -p $acrToken

az acr import `
  --name $acr `
  --source docker.io/appsmith/appsmith-ee:latest `
  --image appsmith/appsmith-ee:latest `
  --username [Docker personal access token username] `
  --password [Docker personal access token] `
  --force

Then when you spin up an image within ACI you have to log into your ACR and specify the same image you created with the import command. This is just a partial piece of the az container create command This uses the admin mode user/pass method

    --image $acrLS/appsmith/appsmith-ee:latest `
    --registry-login-server $acrLS `
    --registry-username [ACR admin username] `
    --registry-password [ACR admin password]

[magliok-wwt]

@magilok-wwt It's two weeks later but I just spent some time wrestling with this bear and can tell you you'll need to add: --registry-username [dockerhub username] --registry-password [dockerhub password] --registry-login-server index.docker.io to your az container create command and it should work fine.

Seems like unauthenticated requests between Azure and DockerHub reflect as the "same" account/ip/whatever and get throttled but the minute you drop in credentials, it works as expected without having to create a new ACR just to pull a public image.

That is what I was looking for - thank you. Maybe I can git rid of my crazy ACR above

[karl-petter-sj]

@magilok-wwt

That is what I was looking for - thank you. Maybe I can git rid of my crazy ACR above

Did you solve it? When I tried:

az container create --registry-username [username] --registry-password [passwd]! --registry-login-server index.docker.io --resource-group rg-test --file test.yaml

I get An error response is received from the docker registry 'index.docker.io'. Please retry later.

[magliok-wwt]

So a couple of things...

We did end up doing our own ACR... However, I have found out how not NOT need docker. I can do everything in the Azure Cloud Shell.

MSFT keeps changing things on us with these docker changes.

At first - within the command for 'az container create'

--registry-username [dockerhub username] 
--registry-password [dockerhub password] 
--registry-login-server index.docker.io

( if you use SSO for dockerhub - once you login you'll get your username from the profile popup window )

Additionally what used to be the magic... isn't anymore You used to have to login to your registry using this weird thing

$acr = "yourACRnamehere"  
$acrToken = $(az acr login --name $acr --expose-token --query "accessToken" -o tsv)
$acrLS = $(az acr show --name $acr --query loginServer --output tsv)
docker login $acrLS -u 00000000-0000-0000-0000-000000000000 -p $acrToken

HOWEVER, this need went away

Now it's like this (using appsmith as an image example)

$acr = "yourACRnamehere"  
az acr import `
  --name $acr `
  --source docker.io/appsmith/appsmith-ee:latest `
  --image appsmith/appsmith-ee:latest `
  --username [Docker personal access token username] `
  --password [Docker personal access token] `
  --force

Then on your container create

$location=""
$resourceGroupName=""
$storageAccountName=""
$fileShareName=""
$vnet=""
$subNet=""
$aciName=""

$storageAccountKey=$(az storage account keys list -g $resourceGroupName -n $storageAccountName --query [0].value -o tsv)

az container create `
    --resource-group $resourceGroupName `
    --name $aciName `
    --vnet $vnet `
    --subnet $subNet `
    --ip-address private `
    --ports 80 443 `
    --cpu 2 `
    --memory 4 `
    --azure-file-volume-account-name $storageAccountName `
    --azure-file-volume-account-key $storageAccountKey `
    --azure-file-volume-share-name $fileShareNameCE `
    --azure-file-volume-mount-path "/appsmith-stacks/" `
    --image $acrLS/appsmith/appsmith-ce `
    --registry-login-server $acrLS `
    --registry-username [ACR-Master-User] `
    --registry-password [ACR-Master-Pass]

yes, yes... I know the storage key in plain-text ¯_(ツ)_/¯

[karl-petter-sj]

My colleague just helped me and came up with a much simpler solution for us! Instead of relying on DockerHub he suggested using the quay.io registry instead. So I just changed image: ubuntu:latest to image: quay.io/bedrock/ubuntu:latest in the container file.

[Siedlerchr]

@magilok-wwt It's two weeks later but I just spent some time wrestling with this bear and can tell you you'll need to add: --registry-username [dockerhub username] --registry-password [dockerhub password] --registry-login-server index.docker.io to your az container create command and it should work fine.

With an image from an org account it works but not from a user account .e.g I am unable to pull this image in azure foppiano/grobid:0.8.1 althoug it exists https://hub.docker.com/r/lfoppiano/grobid/tags

A docker pull on my local machine worked fine... I even tried the stuff with username and password