Open nmatare opened 1 year ago
Which version of Argo CD are you using? Since 2.5.5 (see #11327) this should have been fixed.
Can confirm I'm on 2.5.5
argocd: v2.5.5+fc3eaec.dirty
BuildDate: 2022-12-16T18:35:58Z
GitCommit: fc3eaec6f498ddbe49a5fa9d215a219191fba02f
GitTreeState: dirty
GoVersion: go1.19.4
Compiler: gc
Platform: darwin/arm64
W1228 18:05:10.659529 66528 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
argocd-server: v2.5.5+fc3eaec
BuildDate: 2022-12-16T16:10:26Z
GitCommit: fc3eaec6f498ddbe49a5fa9d215a219191fba02f
GitTreeState: clean
GoVersion: go1.18.9
Compiler: gc
Platform: linux/amd64
Kustomize Version: v4.5.7 2022-08-02T16:35:54Z
Helm Version: v3.10.3+g835b733
Kubectl Version: v0.24.2
Jsonnet Version: v0.18.0
Also sanity checked the service account with the below if its helpful for debugging:
SERVICE_ACCOUNT=$(cat *.json | base64)
cat ${SERVICE_ACCOUNT} | helm registry login -u _json_key_base64 --password-stdin https://us-east1-docker.pkg.dev
>>> Login Succeeded
helm pull oci://us-east1-docker.pkg.dev/$project_id/testing-helm/$chart --version $version
>>> Pulled
Hmm that's strange. Perhaps you're bumping into the same issue I bumped into in #10218?
@nmatare I believe the issue is a mismatch between the repo that you add: us-east1-docker.pkg.dev
and the repo that the application is using: us-east1-docker.pkg.dev/$project/testing-helm
can you try with:
argocd app create testing-application \
--dest-name=in-cluster \
--repo us-east1-docker.pkg.dev \
--helm-chart=$project/testing-helm/${chart} --revision ${version} --helm-pass-credentials
?
Or, if that doesn't work, stick to the current app create, but define the repo like this:
argocd repo add \
us-east1-docker.pkg.dev/$project/testing-helm/ \
--port-forward-namespace argocd \
--insecure-skip-server-verification \
--type helm \
--enable-oci \
--upsert \
--name testing-helm \
--username=_json_key_base64 \
--password=${SERVICE_ACCOUNT}
Thanks @alexef , your first suggestion worked for me. 👍🏻
@nmatare I believe the issue is a mismatch between the repo that you add:
us-east1-docker.pkg.dev
and the repo that the application is using:us-east1-docker.pkg.dev/$project/testing-helm
can you try with:
argocd app create testing-application \ --dest-name=in-cluster \ --repo us-east1-docker.pkg.dev \ --helm-chart=$project/testing-helm/${chart} --revision ${version} --helm-pass-credentials
?
Or, if that doesn't work, stick to the current app create, but define the repo like this:
argocd repo add \ us-east1-docker.pkg.dev/$project/testing-helm/ \ --port-forward-namespace argocd \ --insecure-skip-server-verification \ --type helm \ --enable-oci \ --upsert \ --name testing-helm \ --username=_json_key_base64 \ --password=${SERVICE_ACCOUNT}
@alexef The first option worked! Although it's a tad annoying to include the project ID in the chart name vs in the repo config. Maybe this can be addressed in the future.
The second option doesn't work, repo creation fails as OCI repos should just have a host and port (ie no path)
I am getting the same error.
My configuration is the same except I am using "declarative mode".
I tried both approaches mentioned by @alexef but still no luck :(
Could you see if I am missing anything?
ArgoCD Helm is configured with:
repositories: |
- url: "us-central1-docker.pkg.dev"
name: "My Helm Chart Repository"
enableOCI: true
type: helm
project: abc
username: _json_key_base64
password: "base64 json key"
The root app is configured like this:
- name: root-app
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
project: default
source:
repoURL: us-central1-docker.pkg.dev
chart: project/repository/chart
targetRevision: 0.0.1
helm:
passCredentials: true
error reported by the UI:
rpc error: code = Unknown desc = `helm pull oci://us-central1-docker.pkg.dev/project/repository/chart --version 0.0.1 --destination /tmp/01252c11-c37c-467d-907c-eabf79c716b1`
failed exit status 1: Error: failed to authorize: failed to fetch anonymous token: unexpected status: 403 Forbidden
@alexef thanks for the help! I'm unable to get it to register with either one or two. Here's what I'm seeing:
Option 1:
(base) ➜ git:(main) ✗ argocd app create testing-application \
--dest-name=in-cluster \
--repo us-east1-docker.pkg.dev \
--helm-chart=$project/testing-helm/$chart --revision $revision --helm-pass-credentials
W0106 20:52:54.824789 22235 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
FATA[0002] rpc error: code = InvalidArgument desc = application spec for testing-application is invalid: InvalidSpecError: Unable to generate manifests in : rpc error: code = Unknown desc = `helm pull oci://us-east1-docker.pkg.dev/$project/testing-helm/$chart --version $revision --destination /tmp/f2622a79-bfa5-4bba-b130-6122417223f3` failed exit status 1: Error: us-east1-docker.pkg.dev/$project/testing-helm/$chart:$revision: not found
Option 2:
As @alexstojda hinted, this will fail on the OCI host/port resolution:
(base) ➜ git:(main) ✗ argocd repo add \
us-east1-docker.pkg.dev/$project/testing-helm/ \
--port-forward-namespace argocd \
--insecure-skip-server-verification \
--type helm \
--enable-oci \
--upsert \
--name testing-helm \
--username=_json_key_base64 \
--password=${SERVICE_ACCOUNT}
W0106 20:50:02.636372 21998 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
FATA[0002] rpc error: code = Unknown desc = error testing repository connectivity: `helm registry login us-east1-docker.pkg.dev/$project/testing-helm/ --username ****** --password ****** --insecure` failed exit status 1: WARNING: Using --password ****** the CLI is insecure. Use --password-stdin.
Error: insecure registry us-east1-docker.pkg.dev/$project/testing-helm/ is not valid: invalid host "us-east1-docker.pkg.dev/$project/testing-helm/"
Interestingly, I can add the repo if I do the below, but I get anonymous lookups when I try to use the repo. Probably because it's unaware of the project and chart name. (I've also tried different combinations of us-east1-docker.pkg.dev/$project/$chart
for the first argument.)
(base) ➜ git:(main) ✗ argocd repo add \
us-east1-docker.pkg.dev \
--port-forward-namespace argocd \
--insecure-skip-server-verification \
--type helm \
--enable-oci \
--upsert \
--name testing-helm \
--username=_json_key_base64 \
--password=${SERVICE_ACCOUNT}
W0106 20:57:24.031717 22546 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead.
To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke
Repository 'us-east1-docker.pkg.dev' added
@alexef thanks for the help! I'm unable to get it to register with either one or two. Here's what I'm seeing:
Option 1:
(base) ➜ git:(main) ✗ argocd app create testing-application \ --dest-name=in-cluster \ --repo us-east1-docker.pkg.dev \ --helm-chart=$project/testing-helm/$chart --revision $revision --helm-pass-credentials W0106 20:52:54.824789 22235 gcp.go:120] WARNING: the gcp auth plugin is deprecated in v1.22+, unavailable in v1.25+; use gcloud instead. To learn more, consult https://cloud.google.com/blog/products/containers-kubernetes/kubectl-auth-changes-in-gke FATA[0002] rpc error: code = InvalidArgument desc = application spec for testing-application is invalid: InvalidSpecError: Unable to generate manifests in : rpc error: code = Unknown desc = `helm pull oci://us-east1-docker.pkg.dev/$project/testing-helm/$chart --version $revision --destination /tmp/f2622a79-bfa5-4bba-b130-6122417223f3` failed exit status 1: Error: us-east1-docker.pkg.dev/$project/testing-helm/$chart:$revision: not found
@nmatare: It seems that you have to replace $project
, $chart
and $revision
for your actual values.
@jacek-jablonski, oh those are simply sensitive values. In my shell I replace them with real values.
Ok, I assumed you didn't replace them, sorry.
It seems that you are referring to a chart or version that doesn't exist, but no problems with authentication.
Yeah, it's strange. On my local box, I can pull the chart just fine with:
SERVICE_ACCOUNT=$(cat *.json | base64)
echo "${SERVICE_ACCOUNT}" | helm registry login -u _json_key_base64 --password-stdin https://us-east1-docker.pkg.dev
>>> Login Succeeded
helm pull oci://us-east1-docker.pkg.dev/$project/testing-helm/$chart --version $revision
>>> Pulled
I was able to overcome this problem by using this yaml apiVersion: v1 kind: Secret metadata: labels: argocd.argoproj.io/secret-type: repository name: helm-gcr-oci-decl namespace: argocd type: Opaque data: enableOCI: dHJ1ZQ== name: Base64OfTheName username: X2pzb25fa2V5 password: SomeSuperLongBase64OfTheServiceAcctFileFromGCP url: Base64OfTheUSWEST1Docker/Project/Repo type: aGVsbQ==
I have a strange issue. If I add the OCI repository for my private helm repo (hosted on azure container registry) everything works. However, if I do it using a kubernetes secret, it does synchronize and everything seems the same but then it doesn't work. The list of repositories says its connected but it won't recognize it if I try to deploy a helm chart that has a dependency in this private repository.
Is there anything that the I need to do extra when declaring the username/password in kubernetes secret? Or put differently, is there anything different when adding a repo from the UI?
So i had the same issue: the below code worked for me, but i had to refresh a few times, then i had to do a hard refresh on each and every application and then after it fixed it but then i needed to sync each and every one as well, deleting the application didnt help, the hard refresh helped. hope that helps you. I did try without token but it didnt work, even with managed identity. Unfortunately, a bug in my opinion is still there as it must have a token
---
apiVersion: v1
stringData:
enableOCI: "true"
name: "lib-chart"
password: "TOKEN_HERE"
type: "helm"
url: "repo.azurecr.io/helm"
username: "helm-library-chart-token-name"
kind: Secret
metadata:
labels:
argocd.argoproj.io/secret-type: repository
namespace: argocd
name: lib-chart
type: Opaque
I was able to overcome this problem by using this yaml apiVersion: v1 kind: Secret metadata: labels: argocd.argoproj.io/secret-type: repository name: helm-gcr-oci-decl namespace: argocd type: Opaque data: enableOCI: dHJ1ZQ== name: Base64OfTheName username: X2pzb25fa2V5 password: SomeSuperLongBase64OfTheServiceAcctFileFromGCP url: Base64OfTheUSWEST1Docker/Project/Repo type: aGVsbQ==
I think it's important to note above that you also need your ArgoCD application to be in the right format:
...
template:
metadata:
name: 'example-{{ .environment }}'
spec:
project: example
sources:
- repoURL: <region>-docker.pkg.dev
chart: <gcp-project-name>/charts/hello-world
...
I tried a few versions including oci://<region>-docker.pkg.dev
etc, as well as having the chart path in the repoURL
, which was causing me issues. Feel free to reach out to me if you're still not able to integrate--using this I was able to integrate ArgoCD with a GCP Artifact Repository Helm Chart repo using a serviceaccount json key file.
So i had the same issue: the below code worked for me, but i had to refresh a few times, then i had to do a hard refresh on each and every application and then after it fixed it but then i needed to sync each and every one as well, deleting the application didnt help, the hard refresh helped. hope that helps you. I did try without token but it didnt work, even with managed identity. Unfortunately, a bug in my opinion is still there as it must have a token
--- apiVersion: v1 stringData: enableOCI: "true" name: "lib-chart" password: "TOKEN_HERE" type: "helm" url: "repo.azurecr.io/helm" username: "helm-library-chart-token-name" kind: Secret metadata: labels: argocd.argoproj.io/secret-type: repository namespace: argocd name: lib-chart type: Opaque
@joshbsemperis Can you detail how your application is set up? I have the repo credentials marked as successfully connected, but the application still fails to pull from the repo, giving the 403 error. Seems like it really doesn't want to use the already specified credentials in the repo connections.
For example in my setup, I have the repo URL in the repositories connections to be without the oci://
prefix, it says connection status is successful. However in the application, even though I didn't specify it, the repo it is pulling from has the oci://
prefix. If I add the oci://
prefix to the repo connection, the connection fails.
Nevermind, I had my repo configuration wrong, it works as intended.
I'm unable to create an application that uses a helm private registry as the source. This seems to be related to the discussion here: https://github.com/argoproj/argo-cd/discussions/10644
I've been able to define the repository and get the UI to register successful. From local:
However, the repo server is unable to make use of the credentials.
Indeed, helm isn't making use of the credentials. Apparently, the repo server is calling anonymously.