Open zuccon opened 1 year ago
We are also running into this issue after migrating from GCR to Artifact Registry.
@jannfis This is going to need to be patched anyway for the merge to ArgoCD, and with GCR being sunset in favour of Artifact Registry. If I was to take the time to submit a PR patching this, would it get merged?
Same here with WorkloadIdentity enabled from within GKE cluster:
Permission \"artifactregistry.repositories.downloadArtifacts\" denied on resource
Another pod with gcloud
in it works as expected using the same ServiceAccount.
Ran into the same issue with the latest version of Argo (following this bug). Looking in the logs, the identity is not passed to the Docker-GetTags
request; see empty authenticationInfo
:
authenticationInfo: {
}
authorizationInfo: [
0: {
permission: "artifactregistry.repositories.downloadArtifacts"
resource: "projects/project-id/locations/europe-west2/repositories/my-repo"
}]
requestMetadata: {
callerIp: "gce-internal-ip"
callerSuppliedUserAgent: "Go-http-client/1.1,gzip(gfe)"
}
Also a bit odd to see v2
in the API here, I only see reference to v1 in the docs:
request: {
@type: "type.googleapis.com/google.logging.type.HttpRequest"
requestMethod: "GET"
requestUrl: "/v2/project-id/my-repo/example-chart-dev/tags/list"
}
We've also encountered this issue after switching to Artifacts Registry. Did anyone find a solution to this yet?
Update: My solution was to update the configMap as follows:
- name: Google Artifacts Registry - us-east4
api_url: https://us-east4-docker.pkg.dev
prefix: us-east4-docker.pkg.dev
ping: no
credentials: ext:/app/config/script.sh
credsexpire: 15m
In my case, we use a script for authentication using workload identity, but it should also work if you create a service account key file, package it into a secret and reference it as shown in the documentation.
credentials: secret:foo/bar#creds
Thx @ralphotowo for pointing me to the right answer. Works for me now.
For everyone else using WorkloadIdentity one hint which cost me some hours of investigation and head-on-the-table-moments:
A script getting an token from the metadata api eg. http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token
is returned an access token. What noone (only the GCP docs somewhere hidden ;-) ) tell you is, that the username is oauth2accesstoken
So to be clear:
If using WorkloadIdentity...
... add a script which
... gets the token from the metadata endpoint and
... returns something like echo "oauth2accesstoken:<yourtoken>"
Hope this helps.
Thx @ralphotowo for pointing me to the right answer. Works for me now.
For everyone else using WorkloadIdentity one hint which cost me some hours of investigation and head-on-the-table-moments:
A script getting an token from the metadata api eg.
http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token
is returned an access token. What noone (only the GCP docs somewhere hidden ;-) ) tell you is, that the username isoauth2accesstoken
So to be clear: If using WorkloadIdentity... ... add a script which ... gets the token from the metadata endpoint and ... returns something like
echo "oauth2accesstoken:<yourtoken>"
Hope this helps.
Thanks @jaykay !
This solution seems to work, at least with argocd-image-updater
GCLOUD_CREDS="oauth2accesstoken:$(gcloud auth print-access-token)" argocd-image-updater test europe-docker.pkg.dev/my-repository/my-helm/my-x --credentials env:GCLOUD_CREDS
DEBU[0000] Creating in-cluster Kubernetes client
INFO[0000] retrieving information about image image_alias= image_name=europe-docker.pkg.dev/my-repository/my-helm/my-x registry_url=europe-docker.pkg.dev
DEBU[0000] setting rate limit to 20 requests per second prefix=europe-docker.pkg.dev registry="https://europe-docker.pkg.dev"
DEBU[0000] Inferred registry from prefix europe-docker.pkg.dev to use API https://europe-docker.pkg.dev
INFO[0000] Fetching available tags and metadata from registry application=test image_alias= image_name=europe-docker.pkg.dev/my-repository/my-helm/my-x registry_url=europe-docker.pkg.dev
INFO[0000] Found 21 tags in registry application=test image_alias= image_name=europe-docker.pkg.dev/my-repository/my-helm/my-x registry_url=europe-docker.pkg.dev
DEBU[0000] found 21 from 21 tags eligible for consideration image=europe-docker.pkg.dev/my-repository/my-helm/my-x
INFO[0000] latest image according to constraint is europe-docker.pkg.dev/my-repository/my-helm/kubernator-x:0.0.21 application=test image_alias= image_name=europe-docker.pkg.dev/my-repository/my-helm/my-x registry_url=europe-docker.pkg.dev
Now I need to figure out how to incorporate this into my ArgoCD configuration.
I have the same issue but with helm charts instead of the docker images, so I guess I can not use the argocd-image-updater
as mentioned by @jaykay
I have the same issue but with helm charts instead of the docker images, so I guess I can not use the
argocd-image-updater
as mentioned by @jaykay
I finally seem to have figured it out by debugging the oras-go client. It turns out the provided credentials in the Repository setting where not used because oras-go compares the target with the registry to see if it can use they match. If they do not match EmptyCredential
is removed.
In my case I configured the repository URL as: europe-docker.pkg.dev/my-repository/my-helm
but the target
evaluated to: europe-docker.pkg.dev
.
I had to add the registry without the path: europe-docker.pkg.dev
and in application sources put the full path in the chart
e.g:
- chart: my-repository/my-helm/my-x
repoURL: europe-docker.pkg.dev
Which just feels wrong. Should I create another issue for this? I am guessing it's the same exact case for the argocd-image-updater? I have not checked though.
@stefangluszek Thank you for the details!
Now I need to figure out how to incorporate this into my ArgoCD configuration.
Did you manage to configure ArgoCD Image Updater to make it working with container images (not Helm charts) stored in GAR? Could you please share your solution?
Nevermind, I found the solution here: https://github.com/argoproj-labs/argocd-image-updater/issues/319#issuecomment-1130547057 (thanks to @bkanuka! 🎉 )
In case if ArgoCD Image Updater is installed using a Helm chart, the setup for GAR auth via Workload Identity will look like this (example of your custom helm values for argocd-image-updater chart):
# values.yaml
config:
registries:
- name: Google Artifacts Registry - eu-west1
api_url: https://europe-docker.pkg.dev
prefix: europe-docker.pkg.dev
credentials: ext:/scripts/gke-workload-identity-auth.sh
credsexpire: 30m
authScripts:
enabled: true
scripts:
gke-workload-identity-auth.sh: |
#!/bin/sh
ACCESS_TOKEN=$(wget --header 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token -q -O - | grep -Eo '"access_token":.*?[^\\]",' | cut -d '"' -f 4)
echo "oauth2accesstoken:$ACCESS_TOKEN"
thank you very much @legal90 you save my time.
@legal90 You saved me at least a day's work messing around with this, thank you!
@legal90 Thanks for your solution here. I have added this to my helm chart because I'm trying to get this argocd-image-updater but am still hitting this issue:
denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/t-computer-410410/locations/europe-west4/repositories/boutique-app" (or it may not exist)
When I exec into my argo-cd-image-updater pod I can't see a /scripts directory which I'm assuming the script should get mounted to.
I've checked other things like the permissions on my SA itself and that is set to Artifact Registry Owner, along with the annotation on my GKE SA to map it to my GCP IAM one.
Do I need to mount a volume as mentioned here by @bkanuk: https://github.com/argoproj-labs/argocd-image-updater/issues/319#issuecomment-1130547057 ?
@legal90 Thanks for your solution here. I have added this to my helm chart because I'm trying to get this argocd-image-updater but am still hitting this issue:
denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/t-computer-410410/locations/europe-west4/repositories/boutique-app" (or it may not exist)
When I exec into my argo-cd-image-updater pod I can't see a /scripts directory which I'm assuming the script should get mounted to.
I've checked other things like the permissions on my SA itself and that is set to Artifact Registry Owner, along with the annotation on my GKE SA to map it to my GCP IAM one.
Do I need to mount a volume as mentioned here by @bkanuk: #319 (comment) ?
Figured this out - I had to delete the whole deployment and not just the pod for this to show in the containers file system. However, still getting a permission error when trying to access my Artifact Registry repository. Interestingly, when I run a "gcloud auth print-access-token" command and save the token details, it looks like this (much shorter token and this works when I try to access my registry) ya29.a0AfB_byA_q-IsTFhKXdjs32Q3vMVsQ6v1jw1u0BHr7g9Rz_YrEUt4BSJ6Ssvo3jHKd83ld0Utqrk9zLBSahL2G9KBk4P6S77aVDHIh9-leAgIyScfmpXqhj9d3UDj4UCz3wgy7uczqPBsadasb1nXkAnzXE4gY4VtJe8aCgYKAR4SARESFQHGX2MibS54WHYZ2-QB7nBrrtGwtg0178
But when I use your script to get an access token, it retrieves a much longer token which doesn't work when accessing my registry: ya29.c.c0AY_VpZiDwXg2Rh8co06iVcLjYS463stzOHTHTj3IoSGQDzvTeDAjJuxkghYz_Ss79b5T5y2IN4l1fiBsW6OW0AEThO3AH52Za8N2EWDsCOXgBQ6qjcODspPZsWHfiujPazJgCVcOJuGkVWVQwvryRelgopJyBdH_knOJ4CyVx07pf_trrmW_Nx6GSCRtzNrNmkUNOnTvoYUwUTqBi4oFgtUStqL05cYbHEWrtZ9Z3f1YjnDmYX-lrcwWLbEIUI4gwhBS2W908Ur9pClsacGkREK77NaPK3M_J_7shc1YUtoAITBbk1gL45xnfUgLQEuvS4YhmpdNvrbxNPLJMxy2bQcUVvPaStDrwPoCnxyzZ1gFH9Ngl_a3yBeWIKT1C9u92oH396DB3wwsidJmkbIa17udSslUwV-v2Bo8qYIVjku1sziOJqhrQRhpJI5Rlr22rww-i8cOarqVFf7ntkQO7_66b_5BcUXB3d2zJ3VqcJv71byigsYZrc87bef96jzdX8ee30Iys_mtsJ0cVFiy9IwJZUBYXF1_zmbjb1q165zaROidX2wZlY0jpIxQVXqufWOyjbBO4d-ds81UVcJVkIsiUF0jBnckSXXqaZ9RZ9u24uQjbijz7XtmB7peuFv9YzQ50fqm_xljxJri-yws8BouhFxoetzs56aziw4cM8lhAZoh7kOtj7Fmju9hZXp1-WRleyVZbVRVXQcJk6qR7j3m1dS-fv1hF-gzVcsW97-tSajuh8UbSFv7elxh7Rq0MY9I0sWSqBnXisYJwB2uxndfawj7ahdYOXXcY5wov3IQsZM7--9RyWYW9vtmBah1aoMy71QFvpZIc-5xX3aOJmp_Fo9B6_khZyOkVSkxp8m-UZ1elceBzXsipZXU3l-XXYp5QJi8zf6tp2q8-e70hkmwz89IBXJ8SkjXlsZxOFMUrYWu2o1ZBB9O53ywxrqX8F9Jr1Sc7sVcy08zbjFZmJlbX76cF2k1fSw53jj2eoqivUMMVo7n
I've changed these tokens so they won't work ;) But does anyone know why the shorter token would work, but the longer one would give a permission issue??
@legal90 Thanks for your solution here. I have added this to my helm chart because I'm trying to get this argocd-image-updater but am still hitting this issue: denied: Permission "artifactregistry.repositories.downloadArtifacts" denied on resource "projects/t-computer-410410/locations/europe-west4/repositories/boutique-app" (or it may not exist) When I exec into my argo-cd-image-updater pod I can't see a /scripts directory which I'm assuming the script should get mounted to. I've checked other things like the permissions on my SA itself and that is set to Artifact Registry Owner, along with the annotation on my GKE SA to map it to my GCP IAM one. Do I need to mount a volume as mentioned here by @bkanuk: #319 (comment) ?
Figured this out - I had to delete the whole deployment and not just the pod for this to show in the containers file system. However, still getting a permission error when trying to access my Artifact Registry repository. Interestingly, when I run a "gcloud auth print-access-token" command and save the token details, it looks like this (much shorter token and this works when I try to access my registry) ya29.a0AfB_byA_q-IsTFhKXdjs32Q3vMVsQ6v1jw1u0BHr7g9Rz_YrEUt4BSJ6Ssvo3jHKd83ld0Utqrk9zLBSahL2G9KBk4P6S77aVDHIh9-leAgIyScfmpXqhj9d3UDj4UCz3wgy7uczqPBsadasb1nXkAnzXE4gY4VtJe8aCgYKAR4SARESFQHGX2MibS54WHYZ2-QB7nBrrtGwtg0178
But when I use your script to get an access token, it retrieves a much longer token which doesn't work when accessing my registry: ya29.c.c0AY_VpZiDwXg2Rh8co06iVcLjYS463stzOHTHTj3IoSGQDzvTeDAjJuxkghYz_Ss79b5T5y2IN4l1fiBsW6OW0AEThO3AH52Za8N2EWDsCOXgBQ6qjcODspPZsWHfiujPazJgCVcOJuGkVWVQwvryRelgopJyBdH_knOJ4CyVx07pf_trrmW_Nx6GSCRtzNrNmkUNOnTvoYUwUTqBi4oFgtUStqL05cYbHEWrtZ9Z3f1YjnDmYX-lrcwWLbEIUI4gwhBS2W908Ur9pClsacGkREK77NaPK3M_J_7shc1YUtoAITBbk1gL45xnfUgLQEuvS4YhmpdNvrbxNPLJMxy2bQcUVvPaStDrwPoCnxyzZ1gFH9Ngl_a3yBeWIKT1C9u92oH396DB3wwsidJmkbIa17udSslUwV-v2Bo8qYIVjku1sziOJqhrQRhpJI5Rlr22rww-i8cOarqVFf7ntkQO7_66b_5BcUXB3d2zJ3VqcJv71byigsYZrc87bef96jzdX8ee30Iys_mtsJ0cVFiy9IwJZUBYXF1_zmbjb1q165zaROidX2wZlY0jpIxQVXqufWOyjbBO4d-ds81UVcJVkIsiUF0jBnckSXXqaZ9RZ9u24uQjbijz7XtmB7peuFv9YzQ50fqm_xljxJri-yws8BouhFxoetzs56aziw4cM8lhAZoh7kOtj7Fmju9hZXp1-WRleyVZbVRVXQcJk6qR7j3m1dS-fv1hF-gzVcsW97-tSajuh8UbSFv7elxh7Rq0MY9I0sWSqBnXisYJwB2uxndfawj7ahdYOXXcY5wov3IQsZM7--9RyWYW9vtmBah1aoMy71QFvpZIc-5xX3aOJmp_Fo9B6_khZyOkVSkxp8m-UZ1elceBzXsipZXU3l-XXYp5QJi8zf6tp2q8-e70hkmwz89IBXJ8SkjXlsZxOFMUrYWu2o1ZBB9O53ywxrqX8F9Jr1Sc7sVcy08zbjFZmJlbX76cF2k1fSw53jj2eoqivUMMVo7n
I've changed these tokens so they won't work ;) But does anyone know why the shorter token would work, but the longer one would give a permission issue??
In the script, the command used for token invocation (wget --header 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token) is exploiting an API call. It achieves two things:
On the other hand "gcloud auth print-access-token" command doesn't directly interact with an API endpoint. It utilizes the Google Cloud SDK libraries installed on your local machine to fetch and print an access token. So In essence, there's no single API endpoint involved. The gcloud tool and the local SDK libraries handle the entire process of fetching and printing the access token using established Google Cloud authentication mechanisms.
I hope this sheds some light on the matter.
Example for multiregion us
registries:
- name: GCP Artifact Registry
api_url: https://us-docker.pkg.dev
prefix: us-docker.pkg.dev
credentials: ext:/scripts/gcp-auth.sh # defined in authScripts below
credsexpire: 30m
default: true
...
authScripts:
enabled: true
scripts:
gcp-auth.sh: |
#!/bin/sh
ACCESS_TOKEN=$(wget --header 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token -q -O - | grep -Eo '"access_token":.*?[^\\]",' | cut -d '"' -f 4)
echo "oauth2accesstoken:$ACCESS_TOKEN"
...
# Apart from the above don't forget to bind the GCP service account to the K8s service account in `values.yaml`:
serviceAccount:
create: true
annotations: {
# the GCP service account to bind the present k8s service account
iam.gke.io/gcp-service-account: sa-gcp@my-project.iam.gserviceaccount.com
}
I wrote an article here with more instructions in case someone needs further troubleshooting, I hope it saves you time since I didn't find a comprehensive resource for authenticating argocd-image-updater
in GCP Artifact Registry.
Describe the bug argocd-image-updater is unable to get the tag list from Google Artifact Registry.
Behaviour can be reproduced running the binary locally or on Kubernetes.
Getting the image tags list using the podman CLI ( podman image search --list-tags ) returns the expected results.
To Reproduce Steps to reproduce the behavior:
Expected behavior The tag list is successfully fetched.
Version local: v0.12.2 k8s: quay.io/argoprojlabs/argocd-image-updater:v0.12.0
Logs From the Kubernetes pod: