Unable to authenticate to Azure ACR using repository tokens

[gilles-gosuin]

Describe the bug

argocd-image-updater fails to retrieve tags when registry authentication is configured to use Azure ACR tokens.

To Reproduce

Version 1: Using the token name and password as credentials

That was my initial attempt:

1. Create an ACR repository (Premium SKU is required for tokens)
2. Create a scope map, a token and generate token password(s) (https://docs.microsoft.com/en-us/azure/container-registry/container-registry-repository-scoped-permissions)
3. Configure argocd-image-updater to use the token name and password as credentials for the registry

When I realized it wouldn't work, I attempted it with Service Principal credentials (client id / secret) and it worked fine.

Version 2: Using a credential script to generate an OAuth token using the Docker registry API

Since I could successfully do a simple docker login with the token credentials, I assumed that argocd-image-updater did not implement the OAuth flow correctly, I embarked on trying to understand how the Docker Registry API works and I ended up creating the following credentials script (redacted elements have been replaced by meaningful variable names):

#!/bin/sh
set -e
scope="repository:${repository}:pull,metadata_read"
token_credentials=$(echo -n "${token_name}:${token_password}" | base64 -w0)
response=$(wget --header "Authorization: Basic ${token_credentials}" -O - -q "https://${registry_hostname}/oauth2/token?service=${registry_hostname}&scope=${scope}")
token=$(echo -n "${response}" | sed -rn 's/\{\"access_token\"\:\"(.*)\"\}/\1/p')
echo "00000000-0000-0000-0000-000000000000:${token}"

Expected behavior Both versions presented above should work fine, as they follow the Docker Registry API and reproducing them manually works fine as well.

Additional context

Logging in to docker using the token credentials directly works fine:

docker login ${registry_hostname} -u ${token_name} -p ${token_password}

So does listing the tags using a token generated by the registry's token endpoint (using enssentially the same script as the one shown above):

scope="repository:${repository}:pull,metadata_read"
token_credentials=$(echo -n "${token_name}:${token_password}" | base64 -w0)
response=$(wget --header "Authorization: Basic ${token_credentials}" -O - -q "https://${registry_hostname}/oauth2/token?service=${registry_hostname}&scope=${scope}")
token=$(echo -n "${response}" | sed -rn 's/\{\"access_token\"\:\"(.*)\"\}/\1/p')
auth=$(echo -n "00000000-0000-0000-0000-000000000000:${token}" | base64 -w0)
wget --header "Authorization: Basic $auth" -O - -q "https://${registry_hostname}/v2/${repository}/tags/list"

This successful attempt led me to try the credentials script solution... which failed as well. At this point, I'm out of options, which is why I'm creating this issue.

Version v0.12.0+aee153d

Logs

Version 1

$ export CREDENTIALS="${token_name}:${token_password}"
$ argocd-image-updater test ${registry}.azurecr.io/${repository} --credentials env:CREDENTIALS

time="2022-08-22T17:33:28Z" level=debug msg="Creating in-cluster Kubernetes client"
time="2022-08-22T17:33:28Z" level=info msg="retrieving information about image" image_alias= image_name=${registry}.azurecr.io/${repository} registry_url=${registry}.azurecr.io
time="2022-08-22T17:33:28Z" level=debug msg="setting rate limit to 20 requests per second" prefix=${registry}.azurecr.io registry="https://${registry}.azurecr.io"
time="2022-08-22T17:33:28Z" level=debug msg="Inferred registry from prefix ${registry}.azurecr.io to use API https://${registry}.azurecr.io"
time="2022-08-22T17:33:28Z" level=info msg="Fetching available tags and metadata from registry" application=test image_alias= image_name=${registry}.azurecr.io/${repository} registry_url=${registry}.azurecr.io
time="2022-08-22T17:33:28Z" level=fatal msg="could not get tags: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.\n" application=test image_alias= image_name=${registry}.azurecr.io/${repository} registry_url=${registry}.azurecr.io

Version 2

Credential script is configured with an annotation at the application level (the logs show that the script gets called as expected).

time="2022-08-22T17:14:09Z" level=info msg="Starting image update cycle, considering 1 annotated application(s) for update"
time="2022-08-22T17:14:09Z" level=debug msg="Processing application ${application}"
time="2022-08-22T17:14:09Z" level=debug msg="Considering this image for update" alias=${alias} application=${application} image_name=${repository} image_tag=${tag} registry=${registry}.azurecr.io
time="2022-08-22T17:14:09Z" level=debug msg="Using no version constraint when looking for a new tag" alias=${alias} application=${application} image_name=${repository} image_tag=${tag} registry=${registry}.azurecr.io
time="2022-08-22T17:14:09Z" level=trace msg="Found update strategy latest" image_alias=${alias} image_name=${registry}.azurecr.io/${repository} registry_url=${registry}.azurecr.io
time="2022-08-22T17:14:09Z" level=trace msg="No match annotation found" image_alias=${alias} image_name=${registry}.azurecr.io/${repository} registry_url=${registry}.azurecr.io
time="2022-08-22T17:14:09Z" level=trace msg="No ignore-tags annotation found" image_alias=${alias} image_name=${registry}.azurecr.io/${repository} registry_url=${registry}.azurecr.io
time="2022-08-22T17:14:09Z" level=trace msg="Using runtime platform constraint linux/amd64" image_alias=${alias} image_name=${registry}.azurecr.io/${repository} registry_url=${registry}.azurecr.io
time="2022-08-22T17:14:09Z" level=trace msg="Fetching credentials for registry https://${registry}.azurecr.io"
time="2022-08-22T17:14:09Z" level=info msg=/scripts/${script}.sh dir= execID=1bcde
time="2022-08-22T17:14:10Z" level=trace msg="Performing HTTP GET https://${registry}.azurecr.io/v2/${repository}/tags/list"
time="2022-08-22T17:14:10Z" level=info msg="Processing results: applications=1 images_considered=1 images_skipped=0 images_updated=0 errors=1"
time="2022-08-22T17:14:10Z" level=error msg="Could not get tags from registry: errors:\ndenied: requested access to the resource is denied\nunauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.\n" alias=${alias} application=${application} image_name=${repository} image_tag=${tag} registry=${registry}.azurecr.io

[michalchecinski]

Hello :) Have you been able to authenticate the o ACR using the AAD Service principal from Kubernetes secret? It works when running inside the container with the --credentials env:CREDENTIALS flag. But when not providing the flag, I believe it should get credentials from the secret configured in ConfigMap's registries.conf. But unfortunately, it's not. Have you tried that way of providing auth?

[gilles-gosuin]

As explained above, authenticating with service principals works correctly. The topic of this issue is related to ACR repository tokens.

[gilles-gosuin]

I have similar issue. Basically trying to make ArgoCD-image-updater pick Azure ACR crds (defined in k8 cm) and with annotation in Argo-Application just want to make it work but no luck if see in automated way. BUT if i try via CLI it work fine (registries.conf holds same data as cm see below-argocd-image-updater-config) $ argocd-image-updater test myrepoblablabla.azurecr.io/pocimage --registries-conf-path /app/config/registries.conf

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd-image-updater.argoproj.io/imgalias.pull-secret: pullsecret:argocd/acr
    argocd-image-updater.argoproj.io/image-list: imgalias=myrepoblablabla.azurecr.io/pocimage

$ kubectl get cm argocd-image-updater-config -n argocd -oyaml

apiVersion: v1
kind: ConfigMap
data:
  registries.conf: |
    registries:
    - name: acr
      api_url: https://myrepoblablabla.azurecr.io
      prefix: myrepoblablabla.azurecr.io
      credentials: pullsecret:argocd/acr

secret, cm, ArgoApp, Image-Updater all running in same namespace.

k8 secret holds ACR access via userName and password from AccessKey of ACR.

SO any one got with working in any way for ArgoCD with Azure ACR??

You're using access keys, not scopes and tokens; can you please remove your comment and create a different issue instead, as it will only bring confusion.

[iamraj007]

Agree was thinking of doing that... let me do that in new issue. sorry for deviating from you token issue.

[oterno]

I was able to make argocd-image-updater work with managed identities. What I found is that argocd-image-updater actually requests a token from ACR with the scope specified, so what you need to do in the script is to output the refresh token that you can get by querying the oauth2/exchange endpoint of the ACR. Here is an example script:

#!/bin/sh

# 1. Get AAD Access token
AAD_ACCESS_TOKEN=$(wget --quiet --header="Metadata: true" \
  --output-document - \
  "http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://management.azure.com/&client_id="$AZURE_CLIENT_ID |
  python3 -c "import sys, json; print(json.load(sys.stdin)['access_token'])")

# 2. Get refresh token
ACR_REFRESH_TOKEN=$(wget --quiet --header="Content-Type: application/x-www-form-urlencoded" \
  --post-data="grant_type=access_token&service=$AZURE_CONTAINER_REGISTRY&tenant=$AZURE_TENANT_ID&access_token=$AAD_ACCESS_TOKEN" \
  --output-document - \
  https://$AZURE_CONTAINER_REGISTRY/oauth2/exchange |
  python3 -c "import sys, json; print(json.load(sys.stdin)['refresh_token'])")

# Script output
echo "00000000-0000-0000-0000-000000000000:$ACR_REFRESH_TOKEN"

[cveld]

@oterno do note though it is the kubelet managed identity you are leveraging here if I am correct. It is surely not the workload identity of the service account. For people looking for a script that fetches an access token by workload identity federation, take a look at https://github.com/argoproj-labs/argocd-image-updater/pull/676.

[oterno]

You are probably right. I have modified the script to use workload identities since posting that comment. See below for a script using workload identities.

#!/bin/sh

# 1. Get AAD Access token
AAD_ACCESS_TOKEN=$(wget --quiet --header="Content-Type: application/x-www-form-urlencoded" \
  --post-data="scope=https%3A%2F%2Fmanagement.azure.com%2F.default&client_id=${AZURE_CLIENT_ID}&client_assertion_type=urn%3Aietf%3Aparams%3Aoauth%3Aclient-assertion-type%3Ajwt-bearer&client_assertion=$(cat "${AZURE_FEDERATED_TOKEN_FILE}")&grant_type=client_credentials" \
  --output-document - \
  "https://login.microsoftonline.com/${AZURE_TENANT_ID}/oauth2/v2.0/token" | python3 -c "import sys, json; print(json.load(sys.stdin)['access_token'])")

# 2. Get refresh token
ACR_REFRESH_TOKEN=$(wget --quiet --header="Content-Type: application/x-www-form-urlencoded" \
  --post-data="grant_type=access_token&service=$AZURE_CONTAINER_REGISTRY&tenant=$AZURE_TENANT_ID&access_token=$AAD_ACCESS_TOKEN" \
  --output-document - \
  https://$AZURE_CONTAINER_REGISTRY/oauth2/exchange | python3 -c "import sys, json; print(json.load(sys.stdin)['refresh_token'])")

# Script output
echo "00000000-0000-0000-0000-000000000000:$ACR_REFRESH_TOKEN"