helm dependency build fails on charts from GCP Artifact Registry

[romanvogman]

Describe the bug

• Created a project foo in argocd that points to a github repo
• Creating an app from foo/chart that contains dependency charts from gcp artifact registry in Chart.yaml
• Argocd uses a service account with access to the artifact registry - has artifact registry admin permissions
• App creation fails on 403 forbidden when running helm dependency build

Argocd uses the same service account to connect to other clusters (argocd-fleet-admin) which works fine (managed to install app when only pointing it to a chart in github repo without oci dependencies)

service account annotations in the helm chart:

    controller:
      serviceAccount:
        annotations:
          "iam.gke.io/gcp-service-account": "argocd-fleet-admin@project.iam.gserviceaccount.com"
    server:
      serviceAccount:
        annotations:
          "iam.gke.io/gcp-service-account": "argocd-fleet-admin@project.iam.gserviceaccount.com"

"argocd-fleet-admin@project.iam.gserviceaccount.com" was also added to the artifact registry repo permissions with admin access

no clue what's missing since the SA seems to be in use to access other clusters, and also has the needed permissions to artifact registry

Version

used v2.9 and also tried v2.7.7

Logs

Unable to generate manifests in argocd/dev: rpc error: code = Unknown desc = Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: could not download oci://us-central1-docker.pkg.dev/repo/chart_dir/chart: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

[dev-whoan]

Any Updates? Same error happened to me.

I am using Helm oci chart as sub chart, and trying to deploy it using ArgoCD.

Application chart which uses Helm OCI charts is looks like;

apiVersion: v2
name: my-app
description: A my-app chart for Kubernetes
type: application
version: 0.0.1
appVersion: 0.0.1
dependencies:
  - name: deployment-chart
    alias: my-app
    version: 0.6.0
    appVersion: 0.0.1
    repository: "oci://asia-northeast3-docker.pkg.dev/my-project/my-repo"
    condition: my-app.enabled
  - name: config-chart
    alias: my-app-cm
    version: 0.0.1
    appVersion: 0.0.1
    repository: "oci://asia-northeast3-docker.pkg.dev/my-project/my-repo"
    condition: my-app-cm.enabled

In Google Artifact Registry, my-project/my-repo have 2 chart (image) named deployment-chart and config-chart.

I also try to pull OCI helm chart from Google Artifact Registry.

Using GKE with Workload Identity Providers to set service account for ArgoCD, I set the service account annotations as @romanvogman did. But the same error occured.

I added the repositories to ArgoCD;

apiVersion: v1
kind: Secret
metadata:
  name: my-private-git-repo
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  url: https://github.com/my-github/repo
  username: username
  password: and_password
  insecure: "true" # Ignore validity of server's TLS certificate. Defaults to "false"
  forceHttpBasicAuth: "true" # Skip auth method negotiation and force usage of HTTP basic auth. Defaults to "false"
  enableLfs: "true" # Enable git-lfs for this repository. Defaults to "false"
---
apiVersion: v1
kind: Secret
metadata:
  name: gar-helm-chart
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
  annotations:
    managed-by: argocd.argoproj.io
stringData:
  enableOCI: "true"
  name: helm-chart
  type: helm
  url: asia-northeast3-docker.pkg.dev/my-project/my-repo
---
apiVersion: v1
kind: Secret
metadata:
  name: gar-deployment-chart
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
  annotations:
    managed-by: argocd.argoproj.io
stringData:
  enableOCI: "true"
  name: deployment-chart
  type: helm
  url: asia-northeast3-docker.pkg.dev/my-project/my-repo/deployment-chart
---
apiVersion: v1
kind: Secret
metadata:
  name: gar-config-chart
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
  annotations:
    managed-by: argocd.argoproj.io
stringData:
  enableOCI: "true"
  name: config-chart
  type: helm
  url: asia-northeast3-docker.pkg.dev/my-project/my-repo/config-chart

[dev-whoan]

@romanvogman I just solved this issue adding username and password with created Google Service Account.

You can follow this docs.

here is one example that how I solved;

apiVersion: v1
kind: Secret
metadata:
  name: gar-config-chart
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
  annotations:
    managed-by: argocd.argoproj.io
stringData:
  enableOCI: "true"
  name: config-chart
  type: helm
  url: asia-northeast3-docker.pkg.dev/my-project/my-repo/config-chart
  username: _json_key         # < From here to the EOF is what I did.
data:
  password: BASE64_FROM_SERVICE_ACCOUNT_KEY_JSON