Unable to authenticate to private helm registry

[nmatare]

I'm unable to create an application that uses a helm private registry as the source. This seems to be related to the discussion here: https://github.com/argoproj/argo-cd/discussions/10644

I've been able to define the repository and get the UI to register successful. From local:

SERVICE_ACCOUNT=$(cat account.json | base64)
argocd repo add \
    us-east1-docker.pkg.dev \
    --port-forward-namespace argocd \
    --insecure-skip-server-verification \
    --type helm \
    --enable-oci \
    --upsert \
    --name testing-helm \
    --username=_json_key_base64 \
    --password=${SERVICE_ACCOUNT}

However, the repo server is unable to make use of the credentials.

argocd app create testing-application \
    --dest-name=in-cluster \
    --repo us-east1-docker.pkg.dev/$project/testing-helm \
    --helm-chart=${chart} --revision ${version} --helm-pass-credentials

FATA[0001] rpc error: code = InvalidArgument desc = application spec for XX s invalid: InvalidSpecError: Unable to generate manifests in : rpc error: code = Unknown desc = `helm pull oci://us-east1-docker.pkg.dev/XXX/testing-helm/CHART --version VERSION --destination /tmp/1fe9c540-3ac7-4f14-9a41-17c62e3b014d` failed exit status 1: Error: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

Indeed, helm isn't making use of the credentials. Apparently, the repo server is calling anonymously.

kubectl exec argocd-repo-server-7bd9fd899c-hbwsc -n argocd -it /bin/bash

argocd@argocd-repo-server-7bd9fd899c-hbwsc:~$ helm pull oci://us-east1-docker.pkg.dev/project/testing-helm/chart --version version
Error: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

[blakepettersson]

Which version of Argo CD are you using? Since 2.5.5 (see #11327) this should have been fixed.

[nmatare]

Can confirm I'm on 2.5.5

argocd: v2.5.5+fc3eaec.dirty
  BuildDate: 2022-12-16T18:35:58Z
  GitCommit: fc3eaec6f498ddbe49a5fa9d215a219191fba02f
  GitTreeState: dirty
  GoVersion: go1.19.4
  Compiler: gc
  Platform: darwin/arm64
W1228 18:05:10.659529   66528 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
argocd-server: v2.5.5+fc3eaec
  BuildDate: 2022-12-16T16:10:26Z
  GitCommit: fc3eaec6f498ddbe49a5fa9d215a219191fba02f
  GitTreeState: clean
  GoVersion: go1.18.9
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.7 2022-08-02T16:35:54Z
  Helm Version: v3.10.3+g835b733
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.18.0

Also sanity checked the service account with the below if its helpful for debugging:

SERVICE_ACCOUNT=$(cat *.json | base64)
cat ${SERVICE_ACCOUNT} | helm registry login -u _json_key_base64 --password-stdin https://us-east1-docker.pkg.dev
>>> Login Succeeded
helm pull oci://us-east1-docker.pkg.dev/$project_id/testing-helm/$chart --version $version
>>> Pulled

[blakepettersson]

Hmm that's strange. Perhaps you're bumping into the same issue I bumped into in #10218?

[alexef]

@nmatare I believe the issue is a mismatch between the repo that you add: us-east1-docker.pkg.dev and the repo that the application is using: us-east1-docker.pkg.dev/$project/testing-helm

can you try with:

argocd app create testing-application \
    --dest-name=in-cluster \
    --repo us-east1-docker.pkg.dev \
    --helm-chart=$project/testing-helm/${chart} --revision ${version} --helm-pass-credentials

?

Or, if that doesn't work, stick to the current app create, but define the repo like this:

argocd repo add \
    us-east1-docker.pkg.dev/$project/testing-helm/ \
    --port-forward-namespace argocd \
    --insecure-skip-server-verification \
    --type helm \
    --enable-oci \
    --upsert \
    --name testing-helm \
    --username=_json_key_base64 \
    --password=${SERVICE_ACCOUNT}

[felleg]

Thanks @alexef , your first suggestion worked for me. 👍🏻

[alexstojda]

@nmatare I believe the issue is a mismatch between the repo that you add: us-east1-docker.pkg.dev and the repo that the application is using: us-east1-docker.pkg.dev/$project/testing-helm

can you try with:

argocd app create testing-application \
    --dest-name=in-cluster \
    --repo us-east1-docker.pkg.dev \
    --helm-chart=$project/testing-helm/${chart} --revision ${version} --helm-pass-credentials

?

Or, if that doesn't work, stick to the current app create, but define the repo like this:

argocd repo add \
    us-east1-docker.pkg.dev/$project/testing-helm/ \
    --port-forward-namespace argocd \
    --insecure-skip-server-verification \
    --type helm \
    --enable-oci \
    --upsert \
    --name testing-helm \
    --username=_json_key_base64 \
    --password=${SERVICE_ACCOUNT}

@alexef The first option worked! Although it's a tad annoying to include the project ID in the chart name vs in the repo config. Maybe this can be addressed in the future.

The second option doesn't work, repo creation fails as OCI repos should just have a host and port (ie no path)

[dgnemo]

I am getting the same error.

My configuration is the same except I am using "declarative mode".

I tried both approaches mentioned by @alexef but still no luck :(

Could you see if I am missing anything?

ArgoCD Helm is configured with:

    repositories: |
      - url: "us-central1-docker.pkg.dev"
        name: "My Helm Chart Repository"
        enableOCI: true
        type: helm
        project: abc
        username: _json_key_base64
        password: "base64 json key"

The root app is configured like this:

- name: root-app
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
  project: default
  source:
    repoURL: us-central1-docker.pkg.dev
    chart: project/repository/chart
    targetRevision: 0.0.1
    helm:
      passCredentials: true

error reported by the UI:

rpc error: code = Unknown desc = `helm pull oci://us-central1-docker.pkg.dev/project/repository/chart --version 0.0.1 --destination /tmp/01252c11-c37c-467d-907c-eabf79c716b1`
failed exit status 1: Error: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden

[nmatare]

@alexef thanks for the help! I'm unable to get it to register with either one or two. Here's what I'm seeing:

Option 1:

(base) ➜  git:(main) ✗ argocd app create testing-application \
    --dest-name=in-cluster \
    --repo us-east1-docker.pkg.dev \
    --helm-chart=$project/testing-helm/$chart --revision $revision --helm-pass-credentials

W0106 20:52:54.824789   22235 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
FATA[0002] rpc error: code = InvalidArgument desc = application spec for testing-application is invalid: InvalidSpecError: Unable to generate manifests in : rpc error: code = Unknown desc = `helm pull oci://us-east1-docker.pkg.dev/$project/testing-helm/$chart --version $revision --destination /tmp/f2622a79-bfa5-4bba-b130-6122417223f3` failed exit status 1: Error: us-east1-docker.pkg.dev/$project/testing-helm/$chart:$revision: not found

Option 2:

As @alexstojda hinted, this will fail on the OCI host/port resolution:

(base) ➜  git:(main) ✗ argocd repo add \
    us-east1-docker.pkg.dev/$project/testing-helm/ \
    --port-forward-namespace argocd \
    --insecure-skip-server-verification \
    --type helm \
    --enable-oci \
    --upsert \
    --name testing-helm \
    --username=_json_key_base64 \
    --password=${SERVICE_ACCOUNT}

W0106 20:50:02.636372   21998 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
FATA[0002] rpc error: code = Unknown desc = error testing repository connectivity: `helm registry login us-east1-docker.pkg.dev/$project/testing-helm/ --username ****** --password ****** --insecure` failed exit status 1: WARNING: Using --password ****** the CLI is insecure. Use --password-stdin.
Error: insecure registry us-east1-docker.pkg.dev/$project/testing-helm/ is not valid: invalid host "us-east1-docker.pkg.dev/$project/testing-helm/"

Interestingly, I can add the repo if I do the below, but I get anonymous lookups when I try to use the repo. Probably because it's unaware of the project and chart name. (I've also tried different combinations of us-east1-docker.pkg.dev/$project/$chart for the first argument.)

(base) ➜   git:(main) ✗ argocd repo add \
    us-east1-docker.pkg.dev \
    --port-forward-namespace argocd \
    --insecure-skip-server-verification \
    --type helm \
    --enable-oci \
    --upsert \
    --name testing-helm \
    --username=_json_key_base64 \
    --password=${SERVICE_ACCOUNT}
W0106 20:57:24.031717   22546 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
Repository 'us-east1-docker.pkg.dev' added

[jacek-jablonski]

@alexef thanks for the help! I'm unable to get it to register with either one or two. Here's what I'm seeing:

Option 1:

(base) ➜  git:(main) ✗ argocd app create testing-application \
    --dest-name=in-cluster \
    --repo us-east1-docker.pkg.dev \
    --helm-chart=$project/testing-helm/$chart --revision $revision --helm-pass-credentials

W0106 20:52:54.824789   22235 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
FATA[0002] rpc error: code = InvalidArgument desc = application spec for testing-application is invalid: InvalidSpecError: Unable to generate manifests in : rpc error: code = Unknown desc = `helm pull oci://us-east1-docker.pkg.dev/$project/testing-helm/$chart --version $revision --destination /tmp/f2622a79-bfa5-4bba-b130-6122417223f3` failed exit status 1: Error: us-east1-docker.pkg.dev/$project/testing-helm/$chart:$revision: not found

@nmatare: It seems that you have to replace $project, $chart and $revision for your actual values.

[nmatare]

@jacek-jablonski, oh those are simply sensitive values. In my shell I replace them with real values.

[jacek-jablonski]

Ok, I assumed you didn't replace them, sorry.

It seems that you are referring to a chart or version that doesn't exist, but no problems with authentication.

[nmatare]

Yeah, it's strange. On my local box, I can pull the chart just fine with:

SERVICE_ACCOUNT=$(cat *.json | base64)
echo "${SERVICE_ACCOUNT}" | helm registry login -u _json_key_base64 --password-stdin https://us-east1-docker.pkg.dev
>>> Login Succeeded
helm pull oci://us-east1-docker.pkg.dev/$project/testing-helm/$chart --version $revision
>>> Pulled

[onebione]

I was able to overcome this problem by using this yaml apiVersion: v1 kind: Secret metadata: labels: argocd.argoproj.io/secret-type: repository name: helm-gcr-oci-decl namespace: argocd type: Opaque data: enableOCI: dHJ1ZQ== name: Base64OfTheName username: X2pzb25fa2V5 password: SomeSuperLongBase64OfTheServiceAcctFileFromGCP url: Base64OfTheUSWEST1Docker/Project/Repo type: aGVsbQ==

[UXabre]

I have a strange issue. If I add the OCI repository for my private helm repo (hosted on azure container registry) everything works. However, if I do it using a kubernetes secret, it does synchronize and everything seems the same but then it doesn't work. The list of repositories says its connected but it won't recognize it if I try to deploy a helm chart that has a dependency in this private repository.

Is there anything that the I need to do extra when declaring the username/password in kubernetes secret? Or put differently, is there anything different when adding a repo from the UI?

[joshbsemperis]

So i had the same issue: the below code worked for me, but i had to refresh a few times, then i had to do a hard refresh on each and every application and then after it fixed it but then i needed to sync each and every one as well, deleting the application didnt help, the hard refresh helped. hope that helps you. I did try without token but it didnt work, even with managed identity. Unfortunately, a bug in my opinion is still there as it must have a token

---
apiVersion: v1
stringData:
  enableOCI: "true"
  name: "lib-chart"
  password: "TOKEN_HERE"
  type: "helm"
  url: "repo.azurecr.io/helm"
  username: "helm-library-chart-token-name"
kind: Secret
metadata:
  labels:
    argocd.argoproj.io/secret-type: repository
  namespace: argocd
  name: lib-chart
type: Opaque

[ericbram]

I was able to overcome this problem by using this yaml apiVersion: v1 kind: Secret metadata: labels: argocd.argoproj.io/secret-type: repository name: helm-gcr-oci-decl namespace: argocd type: Opaque data: enableOCI: dHJ1ZQ== name: Base64OfTheName username: X2pzb25fa2V5 password: SomeSuperLongBase64OfTheServiceAcctFileFromGCP url: Base64OfTheUSWEST1Docker/Project/Repo type: aGVsbQ==

I think it's important to note above that you also need your ArgoCD application to be in the right format:

...
template:
    metadata:
      name: 'example-{{ .environment }}'
    spec:
      project: example
      sources:
      - repoURL: <region>-docker.pkg.dev
        chart: <gcp-project-name>/charts/hello-world
...

I tried a few versions including oci://<region>-docker.pkg.dev etc, as well as having the chart path in the repoURL, which was causing me issues. Feel free to reach out to me if you're still not able to integrate--using this I was able to integrate ArgoCD with a GCP Artifact Repository Helm Chart repo using a serviceaccount json key file.

[Nickmman]

So i had the same issue: the below code worked for me, but i had to refresh a few times, then i had to do a hard refresh on each and every application and then after it fixed it but then i needed to sync each and every one as well, deleting the application didnt help, the hard refresh helped. hope that helps you. I did try without token but it didnt work, even with managed identity. Unfortunately, a bug in my opinion is still there as it must have a token

---
apiVersion: v1
stringData:
  enableOCI: "true"
  name: "lib-chart"
  password: "TOKEN_HERE"
  type: "helm"
  url: "repo.azurecr.io/helm"
  username: "helm-library-chart-token-name"
kind: Secret
metadata:
  labels:
    argocd.argoproj.io/secret-type: repository
  namespace: argocd
  name: lib-chart
type: Opaque

@joshbsemperis Can you detail how your application is set up? I have the repo credentials marked as successfully connected, but the application still fails to pull from the repo, giving the 403 error. Seems like it really doesn't want to use the already specified credentials in the repo connections.

For example in my setup, I have the repo URL in the repositories connections to be without the oci:// prefix, it says connection status is successful. However in the application, even though I didn't specify it, the repo it is pulling from has the oci:// prefix. If I add the oci:// prefix to the repo connection, the connection fails.

Nevermind, I had my repo configuration wrong, it works as intended.