Cannot pull images from Azure Container Registry

[matteotumiati]

Describe the bug I cannot pull images that are stored in Azure Container Registry.

To Reproduce

1. Install ArgoCD from scratch
2. Install argocd-image-updater (kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml)
3. Create a Kubernetes secret with credentials for the ACR (access using admin):

kubectl create secret docker-registry acr \
    --namespace argocd \
    --docker-server=<container-registry-name>.azurecr.io \
    --docker-username=<container-registry-name> \
    --docker-password=<password>

4. Edit the ConfigMap argocd-image-updater-config (kubectl edit configmap argocd-image-updater-config -n argocd) to include the section for custom registries:

apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"argocd-image-updater-config","app.kubernetes.io/part-of":"argocd-image-updater"},"name":"argocd-image-updater-config","namespace":"argocd"}}
  creationTimestamp: "2023-04-03T08:12:02Z"
  labels:
    app.kubernetes.io/name: argocd-image-updater-config
    app.kubernetes.io/part-of: argocd-image-updater
  name: argocd-image-updater-config
  namespace: argocd
  resourceVersion: "13173"
  uid: 99218d75-12cf-41cf-b9e4-198fce05d60a
data:
  registries.conf: |
    registries:
    - name: 'azure'
      prefix: <container-registry-name>.azurecr.io
      api_url: https://<container-registry-name>.azurecr.io/
      credentials: pullsecret:argocd/acr
      default: true

It is defined as pullsecret, because the secret created in step 3 has a field .dockerconfigjson in the .data section.

5. Restart the deployment, to make sure the ConfigMap is reloaded (kubectl -n argocd rollout restart deployment argocd-image-updater)

6. Create an empty Helm chart and an application to deploy a simple container from the container registry in Azure. It will fail with Init:ImagePullBackOff.

Expected behavior The image can be pulled successfully.

Additional context Passing credentials to the CLI test command works fine.

argocd-image-updater test <container-registry-name>.azurecr.io/<image>:<tag> --credentials pullsecret:argocd/acr

Version Latest

Logs

argocd-image-updater test <container-registry-name>.azurecr.io/busybox:1.36

time="2023-04-03T15:06:26+02:00" level=debug msg="Creating in-cluster Kubernetes client"
time="2023-04-03T15:06:26+02:00" level=info msg="retrieving information about image" image_alias= image_digest= image_name=<container-registry-name>.azurecr.io/busybox image_tag=1.36 registry_url=<container-registry-name>.azurecr.io
time="2023-04-03T15:06:26+02:00" level=debug msg="setting rate limit to 20 requests per second" prefix=<container-registry-name>.azurecr.io registry="https://<container-registry-name>.azurecr.io"
time="2023-04-03T15:06:26+02:00" level=debug msg="Inferred registry from prefix <container-registry-name>.azurecr.io to use API https://<container-registry-name>.azurecr.io"
time="2023-04-03T15:06:26+02:00" level=info msg="Fetching available tags and metadata from registry" application=test image_alias= image_digest= image_name=<container-registry-name>.azurecr.io/busybox image_tag=1.36 registry_url=<container-registry-name>.azurecr.io
time="2023-04-03T15:06:26+02:00" level=fatal msg="could not get tags: Get \"https://<container-registry-name>.azurecr.io/v2/busybox/tags/list\": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information." application=test image_alias= image_digest= image_name=<container-registry-name>.azurecr.io/busybox image_tag=1.36 registry_url=<container-registry-name>.azurecr.io

[pa-mc]

I can confirm. I have the exact same setup. Got the same error. Seem to only work when specifiying --credentials through subcommand test within the CLI.

/ $ cat /app/config/registries.conf
registries:
- name: ACR paservices
  prefix: azurecr.io
  api_url: https://paservices.azurecr.io
  credentials: pullsecret:argocd/pa-paservices-acr
  default: true

[joeferreirapremera]

have you tried to repo this with a kubelet identity that has Acrpull against the ACR?

[Popeye4242]

Azure Container Registry currently works out of the box for azure kubernetes services. You just have to enable managed identity. Using azurecr when not using aks is probably not a good decision anyway. There is no need to specify any credentials if you are using managed identities.

https://learn.microsoft.com/en-us/azure/aks/use-oidc-issuer https://learn.microsoft.com/en-us/azure/aks/use-managed-identity

[Pionerd]

First of all, if you force the pullSecret using the annotation, it also works. I consider this a workaround.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/image-list: test=<name>.azurecr.io/<image>
    argocd-image-updater.argoproj.io/test.pull-secret: pullsecret:argocd/acr-secret

Secondly, in my humble opinion importing a script to perform all actions necessary for Workload Identity is far from "just have to enable managed identity". I'm a huge fan of Workload Identity and use it in almost all components, but in none of them such a script addition is necessary.

Next to that, it just seems off that even though the credentials are correctly configured, they are only used when forced through the annotation. To me this is simply a bug.

Edit (12-Apr): So after testing with Workload Identity (@etiennetremel Thanks for the docs) I noticed that exactly the same issue as with the pull secret is going on:

  $ argocd-image-updater test <acr_name>.azurecr.io/<image_name>@sha256:<image_sha>
DEBU[0000] Creating in-cluster Kubernetes client
INFO[0000] retrieving information about image            image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io
DEBU[0000] setting rate limit to 20 requests per second  prefix=<acr_name>.azurecr.io registry="https://<acr_name>.azurecr.io"
DEBU[0000] Inferred registry from prefix <acr_name>.azurecr.io to use API https://<acr_name>.azurecr.io
INFO[0000] Fetching available tags and metadata from registry  application=test image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io
FATA[0000] could not get tags: Get "https://<acr_name>.azurecr.io/v2/<image_name>/tags/list": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.  application=test image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io
/ $ cat /app/config/registries.conf
registries:
  - api_url: https://<acr_name>.azurecr.io
    credentials: ext:/scripts/auth.sh
    credsexpire: 1h
    default: true
    name: <acr_name>
    prefix: <acr_name>.azurecr.io
/ $ argocd-image-updater test <acr_name>.azurecr.io/<image_name>@sha256:<image_sha> --credentials ext:/scripts/auth.sh
DEBU[0000] Creating in-cluster Kubernetes client
INFO[0000] retrieving information about image            image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io
DEBU[0000] setting rate limit to 20 requests per second  prefix=<acr_name>.azurecr.io registry="https://<acr_name>.azurecr.io"
DEBU[0000] Inferred registry from prefix <acr_name>.azurecr.io to use API https://<acr_name>.azurecr.io
INFO[0000] /scripts/auth.sh                              dir= execID=8aa46
INFO[0000] Fetching available tags and metadata from registry  application=test image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io
INFO[0000] Found 2 tags in registry                      application=test image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io
DEBU[0000] could not parse input tag dev as semver: Invalid Semantic Version
DEBU[0000] could not parse input tag prd as semver: Invalid Semantic Version
INFO[0000] latest image according to constraint is <acr_name>.azurecr.io/<image_name>@sha256:<image_sha>  application=test image_alias= image_digest="sha256:<image_sha>" image_name=<acr_name>.azurecr.io/<image_name> image_tag= registry_url=<acr_name>.azurecr.io

So basically, the issue remains: the config in the registries.conf is not picked up. If the same credentials are forced (either through the --credentials flag when using argocd-image-updater test or the annotation) everything works.

[mtrin]

Stumbled on this by chance. I remember taking a while to figure out but I made it work with acr without workload ids.

  registries.conf: |
    registries:
    - name: <acrname>
      api_url: https://<acrname>.azurecr.io
      ping: no
      credentials: secret:argocd/<secretname>#creds   
      prefix:  <acrname>.azurecr.io

notice how the secret key is specified with the #creds suffix so the secret should be like

data:
  creds: clientid:secret

notice the creds format is important too

Hope it helps someone. This would be good to clarify in the docs @argoproj-labs

[sysadminz]

Hey, have you had any update on this? I am experiencing the same issue. I have had it working for a while but it quit with authentication errors after I updated the token.

I can login with the token through docker cli but the same token is erroring out with argocd image updater.

[etiennetremel]

@sysadminz refer to these setup instructions, it works for me: https://github.com/argoproj-labs/argocd-image-updater/blob/194a433f7207898132bd5865faa3556db55b210b/docs/configuration/registries.md#configuring-azure-container-registry-with

Btw if any maintainer read this, there is a doc update PR waiting to be reviewed since February... it's a long time for just clarifying the doc on how to use Azure Container Registry with the argocd-image-updater...

[avo-sepp]

Those setup instructions work but only if there is a single ACR involved. It is not able to configure more than one ACR. Which is a step in the right direction, but ultimately we need a complete solution which can authenticate with more than one ACR.