Closed Sahaquielxo closed 2 years ago
Hit this as well, started happening after I bumped from 3.25.0
-> 3.26.12
as a result of the github host keys changing.
@Sahaquielxo confirmed that rolling back to 3.25.0
(but including the new github host keys) fixes the issue.
I face the same error, does rolling back to 3.25.0 is the only solution?
@kcrawley-supernatural Rolled back to 3.25.0 and still same issue, any idea?
@LiorLieberman did you manually specify the hostkeys for the recent google key expiration?
knownHosts:
data:
ssh_known_hosts: |
bitbucket.org ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAubiN81eDcafrgMeLzaFPsw2kNvEcqTKl/VqLat/MaB33pZy0y3rJZtnqwR2qOOvbwKZYKiEO1O6VqNEBxKvJJelCq0dTXWT5pbO2gDXC6h6QDXCaHo6pOHGPUy+YBaGQRGuSusMEASYiWunYN0vCAI8QaXnWMXNMdFP3jHAJH0eDsoiGnLPBlBp4TNm6rYI74nMzgz3B9IikW4WVK+dc8KZJZWYjAuORU3jc1c/NPskD2ASinf8v3xnfXeukU0sJ5N6m5E8VLjObPEO+mN2t/FZTMZLiFqPWc/ALSqnMnnhwrNi2rbfg/rd/IpL8Le3pSBne8+seeFVBoGqzHM9yXw==
github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg=
github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl
github.com ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAq2A7hRGmdnm9tUDbO9IDSwBK6TbQa+PXYPCPy6rbTrTtw7PHkccKrpp0yVhp5HdEIcKr6pLlVDBfOLX9QUsyCOV0wzfjIJNlGEYsdlLJizHhbn2mUjvSAHQqZETYP81eFzLQNnPHt4EVVUh7VfDESU84KezmD5QlWpXLmvU31/yMf+Se8xhHTvKSCZIFImWwoG6mbUoWf9nzpIoaSjB+weqqUUmpaaasXVal72J+UX2B+2RPW3RcT0eOzQgqlJL3RKrTJvdsjE3JEAvGq3lGHSZXy28G3skua2SmVi/w4yCE6gbODqnTWlg7+wC604ydGXA8VJiS5ap43JXiUFFAaQ==
gitlab.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFSMqzJeV9rUzU4kWitGjeR4PWSa29SPqJ1fVkhtj3Hw9xjLVXVYrU9QlYWrOLXBpQ6KWjbjTDTdDkoohFzgbEY=
gitlab.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAfuCHKVTjquxvt6CM6tdG4SLp1Btn/nOeHHE5UOzRdf
gitlab.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsj2bNKTBSpIYDEGk9KxsGh3mySTRgMtXL583qmBpzeQ+jqCMRgBqB98u3z++J1sKlXHWfM9dyhSevkMwSbhoR8XIq/U0tCNyokEi/ueaBMCvbcTHhO7FcwzY92WK4Yt0aGROY5qX2UKSeOvuP4D6TPqKF1onrSzH9bx9XUf2lEdWT/ia1NEKjunUqu1xOB/StKDHMoX4/OKyIzuS0q/T1zOATthvasJFoPrAjkohTyaDUz2LN5JoH839hViyEG82yB+MjcFV5MU3N1l1QL3cVUCh93xSaua1N85qivl+siMkPGbO5xR/En4iEY6K2XPASUEMaieWVNTRCtJ4S8H+9
ssh.dev.azure.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H
vs-ssh.visualstudio.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Hr1oTWqNqOlzGJOfGJ4NakVyIzf1rXYd4d7wo6jBlkLvCA4odBlL0mDUyZ0/QUfTTqeu+tm22gOsv+VrVTMk6vwRU75gY/y9ut5Mb3bR5BV58dKXyq9A9UeB5Cakehn5Zgm6x1mKoVyf+FFn26iYqXJRgzIZZcZ5V6hrE0Qg39kZm4az48o0AUbf6Sp4SLdvnuMa2sVNwHBboS7EJkm57XQPVU3/QpyNLHbWDdzwtrlS+ez30S3AdYhLKEOxAG8weOnyrtLJAUen9mTkol8oII1edf7mWWbWVf0nBmly21+nZcmCTISQBtdcyPaEno7fFQMDD26/s0lfKob4Kw8H
@kcrawley-supernatural I did, it is still failing on helm repo add 401 unauthorized when using chart as dependency as described in the issue Are you able to use charts dependencies with helm repos?
any updates ?
After upgrade to v2.2.1 the issue was gone.
The next feature was added to v2.2.0
feat: Update to Helm v3.7.1, allow to pass credentials and new OCI support (#7249)
Also, the CRDs for Application and AppProject were updated in the corresponding Helm Chart version 3.29.0.
If after the upgrade the issue still persist try
argocd app get APPNAME --hard-refresh
Thank you @mcjhknauf Are you sure it solves the issue? I have updated to 3.29.1 and still have the same problem
You are welcome.
At least, that is what I did. I'm using helm to install ArgoCD, so I needed to update the CRDs directly.
After that the issue was still present until I run argocd app get APPNAME --hard-refresh
.
Also, I needed to add the repository credentials with argocd repo add ...
I use helm to install argo as well.
for the app itself I am using umbrella charts - so the helm repo is a dependency.
I have the helm repo is secret - and see that it is green in the settings/repositories screen.
However when the app is still on error status with helm repo add 401 Unauthorized
tried argocd app get APPNAME --hard-refresh
through the UI and it did not help
any other ideas?
for the app itself I am using umbrella charts - so the helm repo is a dependency.
That is my case precisely.
I have the helm repo is secret - and see that it is green in the settings/repositories screen. However when the app is still on error status with
helm repo add 401 Unauthorized
I added the repository initially with the argocd
cli, e.g.:
argocd repo add example.com --type helm --name example --enable-oci --username <username> --password <passwowd>
tried
argocd app get APPNAME --hard-refresh
through the UI and it did not help
in my case I'm using the argocd
cli for all the operations
any other ideas?
Did you add the passCredentials to the Application declaration?
https://github.com/argoproj/argo-cd/blob/v2.2.1/docs/user-guide/helm.md#helm---pass-credentials
spec:
source:
helm:
passCredentials: true
Sorry, I forget to comment about that.
I did and then it showed me the same error, just with --pass-credentials
so it was helm repo add --pass-credentials.....401
Will try through the cli now, does --enable-oci
necessary ?
I did and then it showed me the same error, just with
--pass-credentials
so it washelm repo add --pass-credentials.....401
--pass-credentials is an option for the application only
declarative approach:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: example
namespace: argocd
spec:
project: example
source:
repoURL: example.com
targetRevision: HEAD
path: charts/example
helm:
passCredentials: true
valueFiles:
- values.yaml
destination:
server: https://kubernetes.default.svc
namespace: example
syncPolicy:
automated:
prune: true
from the argocd
cli:
argocd app set APPNAME --helm-pass-credentials
Will try through the cli now, does
--enable-oci
necessary ?
only if a Docker registry is used to store the Helm Charts, for a HTTP based one is not needed
@LiorLieberman I believe I found a couple of related bugs in the way the repository server handles the credentials for dependency charts. Let me ask you to confirm if the following points describe your issue. I'll link later from the bug report once I complete writing, but want to collect related issue (the fix is simple and I will do that too, most likely over the weekend)
From what you said:
If that is the case, the issue is that at this point, the code will only add credentials for repositories that were created on the fly, but not to existing one. So if you could ensure the repository has the proper credentials, that'd help me confirm this. Please share ArgoCD version.
We are having a similar issue. We added a private Azure container registry as a repository in Argo using the following secret:
apiVersion: v1
kind: Secret
metadata:
name: ...
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
type: Opaque
stringData:
url: <subdomain>.azurecr.io
type: 'helm'
enableOci: 'true'
username: <username>
password: <password>
When we try to install a helm chart from the container registry directly as follows, it works:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ...
namespace: argocd
spec:
destination:
namespace: ...
server: 'https://kubernetes.default.svc'
source:
repoURL: '<subdomain>.azurecr.io'
targetRevision: '135830'
chart: '<chart-name>'
helm:
...
However, when we reference a local helm chart which has a dependency on the 'actual' helm chart in the container registry, we get the following error in argo:
rpc error: code = Unknown desc = Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: could not download oci://<subdomain>.azurecr.io/helm/<chart-name>: failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized
This is the application resource which references the local helm chart:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: ...
namespace: argocd
spec:
destination:
namespace: ...
server: 'https://kubernetes.default.svc'
source:
path: <path-to-folder-containing-chart>
repoURL: >-
<url-to-this-repository>
targetRevision: HEAD
helm:
...
And this is the Chart.yaml in the <path-to-folder-containing-chart>
:
apiVersion: v2
name: <chart-name>-wrapper
description: Wrapper around <chart-name>
type: application
version: 1.0.0
appVersion: "1.0.0"
dependencies:
- name: helm/<chart-name>
version: "135830"
repository: oci://<subdomain>.azurecr.io
What we tried so far:
oci://
in the dependencies list of the chart.
This causes the following error: rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: no repository definition for <subdomain>.azurecr.io. Please add them via 'helm repo add' Note that repositories must be URLs or aliases. For example, to refer to the "example" repository, use "https://charts.example.com/" or "@example" instead of "example". Don't forget to add the repo, too ('helm repo add').
passCredentials
property of the application resource to true
As mentioned above I experienced the issue initially, then updated the CRDs to get the passCredentials
present and was solved.
Recently (02/02/2022) some coworkers start to experience the same. At the end, after comparing our solutions the difference was the sourceRepos
in the AppProject
.
In my case I have a "*", but if some repository is already present, the OCI one need to be added. After adding it the issue gone.
Hope this help!
We are having a similar issue. We added a private Azure container registry as a repository in Argo using the following secret:
If you did not have a secret with username and password I'd be tempted to say it might be related to an actual issue I'll report in a moment... but since you do have a repository secret with credentials, please see below.
What we tried so far:
1. Omit `oci://` in the dependencies list of the chart. This causes the following error: `` rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: no repository definition for <subdomain>.azurecr.io. Please add them via 'helm repo add' Note that repositories must be URLs or aliases. For example, to refer to the "example" repository, use "https://charts.example.com/" or "@example" instead of "example". Don't forget to add the repo, too ('helm repo add'). `` 2. Set the `passCredentials` property of the application resource to `true`
@mamjong Yes, you can't omit oci://
from your dependency, helm needs that know it needs to use an OCI registry.
On the passCredentials flag with helm, it does not appear you need it here.
Could you try what @mcjhknauf shared above? The repositories and credentials available to an application depend on project configuration and whether the repo or credentials are allowed. To be sure you can use an asterisk or add multiple sources (and you can asterik along your sources for matching, ArgoCD uses https://github.com/gobwas/glob).
In my case I have a "*", but if some repository is already present, the OCI one need to be added.
@mcjhknauf @ocraviotto We have the following project configuration:
As you can see we have the asterisk wildcard for the source and scoped repositories.
Another thing I did (maybe differently) was creating the Secret directly (this create the repository if contains the proper label). I don't know if related but we observed that at some point the secret name and argocd repo NAME in the repo list were not the same.
Other that that the approach I'm following is install ArgoCD with Terraform using the Helm Provider, then creating the secret for the OCI repository using the Kubernetes Provider and finally installing another Helm Char for an "app of apps" to do the bootstrapping. I tested this locally twice yesterday with minikube and works well.
However, when we reference a local helm chart which has a dependency on the 'actual' helm chart in the container registry, we get the following error in argo:
rpc error: code = Unknown desc = Manifest generation error (cached): `helm dependency build` failed exit status 1: Error: could not download oci://<subdomain>.azurecr.io/helm/<chart-name>: failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized
In the above error you could see Manifest generation error (cached)
, if you try with argocd app get <app name> --hard-refresh
this will force a cleanup of the cache.
For example, during the tests I removed the repository and with the above I get a similar error, but without the Manifest generation error (cached)
part.
ComparisonError rpc error: code = Unknown desc =
helm dependency buildfailed exit status 1: Error: could not download oci://example.com/namespace/apps: failed to authorize: failed to fetch anonymous token: unexpected status: 401 Unauthorized 2022-02-02 18:43:53 +0100 CET
Another thing I did (maybe differently) was creating the Secret directly (this create the repository if contains the proper label). I don't know if related but we observed that at some point the secret name and argocd repo NAME in the repo list were not the same.
Other that that the approach I'm following is install ArgoCD with Terraform using the Helm Provider, then creating the secret for the OCI repository using the Kubernetes Provider and finally installing another Helm Char for an "app of apps" to do the bootstrapping. I tested this locally twice yesterday with minikube and works well.
Thanks for the help. We added a name to the secret as follows:
apiVersion: v1
kind: Secret
metadata:
name: ...
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
type: Opaque
stringData:
name: <subdomain>.azurecr.io
url: <subdomain>.azurecr.io
type: 'helm'
enableOci: 'true'
username: <username>
password: <password>
It doesn't work yet but we do have a different error:
rpc error: code = Unknown desc = `helm repo add --username ****** --password ****** --pass-credentials <subdomain>.azurecr.io <subdomain>.azurecr.io` failed exit status 1: Error: could not find protocol handler for:
Unfortunately the error stops there...
I'm using something like
apiVersion: v1
kind: Secret
metadata:
name: oci-repository
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
type: Opaque
stringData:
name: oci-repository
url: <subdomain>.azurecr.io
type: 'helm'
enableOCI: 'true'
username: <username>
password: <password>
We now added a name
property to the stringData
of the secret which is the same as the name of the secret itself, and it works!
Thank you very much!
We now added a
name
property to thestringData
of the secret which is the same as the name of the secret itself, and it works! Thank you very much!
Great, You're welcome.
We now added a
name
property to thestringData
of the secret which is the same as the name of the secret itself, and it works! Thank you very much!
Correction: The name
value does not necessarily have to be the same as the secret name. Simply adding the name
key-value to the stringData
was enough to fix our problem.
I'm assuming this issue can be closed? Please feel free to re-open if issue still persists.
I'm using something like
apiVersion: v1 kind: Secret metadata: name: oci-repository namespace: argocd labels: argocd.argoproj.io/secret-type: repository type: Opaque stringData: name: oci-repository url: <subdomain>.azurecr.io type: 'helm' enableOCI: 'true' username: <username> password: <password>
we use AWS for microservices, what URL works with this secret key file, also the username and password are for the git repository?
we have created our repository credentials for gitlab using terraform and whilst we are setting the --pass-credentials
in the application, we are still getting the 401.
Like on #7969 we are seeing the helm add -pass-credentials https://repo https://repo
it isn't clear what else we can do... should the same repository credentials for git be passed to helm?
I'm also experiencing this issue.
Argocd version: 2.2.5
My repository secret is as follows:
apiVersion: v1
kind: Secret
metadata:
name: helm-chart-repo
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
stringData:
name: mydomain
url: https://charts.mydomain.com
type: helm
username: blah
password: blah
my Chart.yaml is as follows
apiVersion: v2
name: project
type: application
version: 0.0.0
dependencies:
- name: project
version: 1.0.0
repository: https://charts.mydomain.com
and my argo app:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: project-demo
namespace: argocd
spec:
destination:
server: "https://kubernetes.default.svc"
source:
path: clusters/project
repoURL: ssh://git@github.com/charts.git
targetRevision: master
project: project
When deploying with argo I get this error:
rpc error: code = Unknown desc = helm repo add https://charts.mydomain.com https://charts.mydomain.com failed exit status 1: Error: looks like "https://charts.mydomain.com" is not a valid chart repository or cannot be reached: failed to fetch https://charts.mydomain.com/index.yaml : 401 Unauthorized
Can this issue be reopened?
2.2.5 as well, still getting 401...
edit: tried with both old-style Helm repository and OCI one. Result is the same.
Same issue with ACR and subcharts
Same issue with v2.3.3, ACR and subcharts.
Edit: It works on v2.2.2
Same with v2.1.6, subcharts and artifactory
I found the error. As often Layer 8 ;) The helm repo wasn't allowed in the project as a source repo 🤦♂️ https://github.com/argoproj/argo-cd/issues/7757
Anyone from team?
Hi,
Anyone found a working solution (except downgrade :) )? We have migrated from 2.2.1 to 2.3.3 and private dependencies stopped to work:
ComparisonError rpc error: code = Unknown desc = `helm dependency build` failed exit status 1: Error: could not retrieve list of tags for repository oci:// : GET "https://tags/list": GET "https://": unexpected status code 401: 2022-05-12 09:45:00 +0200 CEST
@WojtekTomaszewski creating other than default
project and binding secrets there worked to me.
@er1z Thanks. We don't use default project at all. Repositories are created under team project and whitelisted in rbac (all, including dependency repos). Tried '*'
for project sourceRepos
but no luck too. Tried all 2.3.x versions with the same result. Rolling back to 2.2.8 did the job. Also gave a shot to 2.4.0 rc and it seems to work again. Confusing...
Started having this issue when we moved a target under its specific Project rather than default
. The only solution was to move it back under default
. Happens in 2.4.11.
Later edit: It seems to also work when specifying each source repo explicitly in the project, rather than using '*'.
I found this issue in Argo v2.6.7 with Project and RBAC configured.
My 2c in case it helps:
kind: Secret
apiVersion: v1
metadata:
name: repo-helm
namespace: argocd
labels:
argocd.argoproj.io/secret-type: repository
data:
name: repo-helm | base64
password: secretPassword | base64
project: Argo Project | base64
type: helm | base64
url: https://$URL/ | base64
username: botusername | base64
type: Opaque
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
.....
sourceRepos:
- >-
https://$URL/
...
If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.
Checklist:
argocd version
.Describe the bug
Even if you add your private helm repository in repositories, argocd will try to add repo without passing credentials. I also tried to add repo manually from repo-server pod, doesn't work.
My configuration:
To Reproduce
repo-server trying to add private helm repo passing user/pass credentials described in the UI.
Screenshots
Version
Logs