Cannot deploy multiple helm dependencies from same private repo server : repository name already exists

romachalm commented 3 years ago

If you are trying to resolve an environment-specific issue or have a one-off question about the edge case that does not require a feature then please consider asking a question in argocd slack channel.

Checklist:

[x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
[x] I've included steps to reproduce the bug.
[x] I've pasted the output of argocd version.

Describe the bug

Argocd 2.1.0 fails at deploying helm chart with dependencies fetched from same private help repository server. It triggers the error :

ComparisonError: rpc error: code = Unknown desc = Manifest generation error (cached): `helm repo add --username ****** --password ****** registry.xxx.io https://registry.xxx.io/chartrepo/cloud/` failed exit status 1: Error: repository name (registry.xxx.io) already exists, please specify a different name

It appears from code that the each new repo using repo-creds will be named as the host. So, if several repositories use the same host, the code tries to add a helm repo with an existing name.

This is a regression from 2.0.5 as this chart was deployed correctly with repository.credentials from argocd-cm

To Reproduce

I have a chart defined as following :

apiVersion: v2
name: my-portal
appVersion: 0.1.0
version: 0.1.0

dependencies:
  - name: admin-portal-web
    version: 1.0.0
    repository: https://registry.xxx.io/chartrepo/enterprise/
    condition: admin-portal-web.enabled

  - name: app-global-cert-manager
    version: 0.1.0
    repository: https://registry.xxx.io/chartrepo/cloud/
    condition: app-global-cert-manager.enabled

Both dependencies comes from the same private server, here harbor, ie chartmuseum. I have configured the repo-creds secret.

apiVersion: v1
kind: Secret
metadata:
  name: harbor-xxx-creds
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repo-creds
data:
  password: xxx=
  username: xxx==
stringData:
  url: https://registry.xxx.io
  type: helm

By parsing the Chart.yaml, the code will add a helm repo named registry.xxx.io for registry.xxx.io/chartrepo/enterprise. Then. for the second depedency, it will try to add the repo registry.xxx.io/chartrepo/cloud which will also be named registry.xxx.io https://github.com/argoproj/argo-cd/blob/a894d4b128c724129752bac9971c903ab6c650ba/reposerver/repository/repository.go#L493

Helm therefore will fire the error Error: repository name (registry.xxx.io) already exists

Expected behavior

This helm chart was correctly deployed with 2.0.5. It should still be deployed with 2.1.0.

Screenshots

n/a

Version

{
    "Version": "v2.1.0+d0b2d55",
    "BuildDate": "2021-08-20T05:30:54Z",
    "GitCommit": "d0b2d55e3fb7fe8b17385d6687886de41651f31b",
    "GitTreeState": "clean",
    "GoVersion": "go1.16.5",
    "Compiler": "gc",
    "Platform": "linux/amd64",
    "KsonnetVersion": "v0.13.1",
    "KustomizeVersion": "v4.2.0 2021-06-30T22:49:26Z",
    "HelmVersion": "v3.6.0+g7f2df64",
    "KubectlVersion": "v0.21.0",
    "JsonnetVersion": "v0.17.0"
}

Logs

argocd-repo-server-67f7f6fd9-796zd argocd-repo-server time="2021-08-25T06:42:21Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** registry.xxx.io https://registry.xxx.io/chartrepo/cloud/` failed exit status 1: Error: repository name (registry.xxx.io) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2021-08-25T06:43:21Z" grpc.service=repository.RepoServerService grpc.start_time="2021-08-25T06:42:21Z" grpc.time_ms=20.81 span.kind=server system=grpc

am250360 commented 3 years ago

I'm also seeing the same thing for deploying dependency charts from multiple public repos with the same host. e.g. for Kyverno plus its policy-reporter:

apiVersion: v2
name: kyverno-deployment
version: v2.0
description: Kyverno Kubernetes Policy Management Deployment
dependencies:
- name: kyverno
  version: v2.0
  repository: "https://kyverno.github.io/kyverno/"
- name: policy-reporter
  version: 1.8.6
  repository: "https://kyverno.github.io/policy-reporter/"

Gives an error log from ArgoCD when syncing:

rpc error: code = Unknown desc = Manifest generation error (cached): `helm repo add kyverno.github.io https://kyverno.github.io/policy-reporter/` failed exit status 1: Error: repository name (kyverno.github.io) already exists, please specify a different name

Should be easy enough to work around (just deploying the two dependency charts in two separate Argo apps), but non ideal.

alexmt commented 3 years ago

Fixed by https://github.com/argoproj/argo-cd/pull/7096

stieglma commented 2 years ago

I am on the newest version (2.3.3) of ArgoCD and I experience more or less the same problem. We have a (test) cluster where I deployed a helm repo secret, and then there's a single reference to the private repository within a helm-chart dependency.

Until a few days ago everything worked fine and then this issue appeared:

rpc error: code = Unknown desc = Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name

I then deleted the argocd-repo-server pod, and everything seemed to be fine again, however an hour later, the same issue appeared again, this time even effecting the build of other helm-based argocd apps, even though they don't reference the private helm repository. Another deletion of argocd-repo-server and everything was fine again for a few minutes. Then only the one test app referencing the private helm repo was back to failing with the above error message again

here are the relevant argocd-repo-server logs. It seems as if the helm-repo-add command is executed multiple times, maybe once for each (helm) app (even though most of them don't use it?), it also seems as if from time to time the helm-repo add command succeeds. I am not sure what is the difference between the failing and the succeeding calls

time="2022-04-08T10:36:21Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T10:37:21Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T10:36:21Z" grpc.time_ms=5.115 span.kind=server system=grpc
time="2022-04-08T10:30:21Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T10:31:21Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T10:30:21Z" grpc.time_ms=9.366 span.kind=server system=grpc
time="2022-04-08T10:27:20Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T10:28:20Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T10:27:20Z" grpc.time_ms=20.363 span.kind=server system=grpc
time="2022-04-08T10:24:20Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T10:25:20Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T10:24:20Z" grpc.time_ms=16.874 span.kind=server system=grpc
time="2022-04-08T10:18:21Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T10:19:21Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T10:18:21Z" grpc.time_ms=9.218 span.kind=server system=grpc
time="2022-04-08T10:15:20Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T10:16:20Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T10:15:20Z" grpc.time_ms=20.297 span.kind=server system=grpc
time="2022-04-08T10:12:20Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T10:13:20Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T10:12:20Z" grpc.time_ms=116.384 span.kind=server system=grpc
time="2022-04-08T10:09:20Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T10:10:20Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T10:09:20Z" grpc.time_ms=15.324 span.kind=server system=grpc
time="2022-04-08T10:06:21Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T10:07:20Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T10:06:20Z" grpc.time_ms=69.943 span.kind=server system=grpc
time="2022-04-08T10:03:20Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T10:04:20Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T10:03:20Z" grpc.time_ms=7.515 span.kind=server system=grpc
time="2022-04-08T10:00:20Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T10:01:20Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T10:00:20Z" grpc.time_ms=5.292 span.kind=server system=grpc
time="2022-04-08T09:57:20Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T09:58:20Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T09:57:20Z" grpc.time_ms=22.771 span.kind=server system=grpc
time="2022-04-08T09:54:20Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T09:55:20Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T09:54:20Z" grpc.time_ms=7.788 span.kind=server system=grpc
time="2022-04-08T09:51:21Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T09:52:21Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T09:51:21Z" grpc.time_ms=7.361 span.kind=server system=grpc
time="2022-04-08T09:48:20Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T09:49:20Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T09:48:20Z" grpc.time_ms=117.342 span.kind=server system=grpc
time="2022-04-08T09:45:21Z" level=error msg="finished unary call with code Unknown" error="`helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T09:46:20Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T09:45:20Z" grpc.time_ms=696.523 span.kind=server system=grpc
time="2022-04-08T09:45:21Z" level=info msg=Trace args="[helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url]" dir=/tmp/_argocd-repo/9be3022a-9fab-488e-b1af-9f9ed3a28321/helm-generic-chart-test operation_name="exec helm" time_ms=57.800124000000004
time="2022-04-08T09:45:21Z" level=error msg="`helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" execID=161a2
time="2022-04-08T09:45:21Z" level=info msg="helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url" dir=/tmp/_argocd-repo/9be3022a-9fab-488e-b1af-9f9ed3a28321/helm-generic-chart-test execID=161a2
time="2022-04-08T09:42:21Z" level=error msg="finished unary call with code Unknown" error="`helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" grpc.code=Unknown grpc.method=GenerateManifest grpc.request.deadline="2022-04-08T09:43:20Z" grpc.service=repository.RepoServerService grpc.start_time="2022-04-08T09:42:20Z" grpc.time_ms=730.961 span.kind=server system=grpc
time="2022-04-08T09:42:21Z" level=info msg=Trace args="[helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url]" dir=/tmp/_argocd-repo/9be3022a-9fab-488e-b1af-9f9ed3a28321/helm-generic-chart-test operation_name="exec helm" time_ms=59.921074
time="2022-04-08T09:42:21Z" level=error msg="`helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url` failed exit status 1: Error: repository name (helm-custom-default) already exists, please specify a different name" execID=d0420
time="2022-04-08T09:42:21Z" level=info msg="helm repo add --username ****** --password ****** --pass-credentials helm-custom-default https://unimportant/repo/url" dir=/tmp/_argocd-repo/9be3022a-9fab-488e-b1af-9f9ed3a28321/helm-generic-chart-test execID=d0420

Update:

I found another way to at least work around the issue:

kubectl exec into the argocd-repo-server pod
helm repo remove helm-custom-default
in the failing argocd app click on force refresh, and then sync

While I played around with that, I noticed that within the repo server the directory layout looked like this:

/helm-working-dir
  repositories.lock
  repositories.yaml
  repository/
    helm-custom-default.txt
    helm-custom-default-index.yaml
    https:/     (yes there is really a folder called https:
      charts.gitlab.io-charts.txt
      ... (all other charts from public repositories, but nothing from the private repo)

is it ok that the folder is called https:? or do I need to reference the charts in a different way?

stieglma commented 2 years ago

@alexmt can you reopen this issue, or should I create a new one with the description posted above? The problem arises every few hours for me, and is then fixed by the workaround I posted, until it fails again.

crenshaw-dev commented 2 years ago

@stieglma can you try to narrow down the variables? Sounds like the potentially-relevant variables are: 1) Multiple apps use the same repo 2) Sometimes manifest generation doesn't fail, maybe due to a new commit clearing a cache? 3) There's a directory called https:

Maybe you could clone the source repo, point a few apps at it, and try to reproduce the problem. Then push a commit and see if you can get one of the apps to generate manifests successfully.

I'm not at all sure, but I suspect that maybe multiple apps are sharing the same build directory on repo-server, and Helm is doing stuff that causes a conflict. Maybe if more than one app weren't generating manifests at the same time, things would work.

stieglma commented 2 years ago

If it helps you to understand the issue better I am happy to schedule a video/screenshare call somewhere next week, but for now a little more context here:

Our environment is a cluster which is completely managed by ArgoCD (it also manages itself). We have one "admin" gitops repository with all the infrastructure setups, and then several referenced "project" gitops repositories (one for each team). In each project gitops repository, there are folders, and each folder is referenced by an app in the admin gitops repository. Helm Charts are never directly configured in the ArgoCD app, but always with a Chart.yaml and a values.yaml + maybe some files in the templates folder. The "base" chart we want to deploy is referenced as a dependency via the Chart.yaml

Overall we have:

6 apps from project repo 1
4 apps from project repo 2
8 apps from project repo 3
33 apps from the admin gitops repo

"Admin" GitOps Repo

.
├── argocd
│   ├── devops
│   │   ├── app-argocd.yaml -> an app referencing the 'argocd' folder (and thus itself) (kustomize resources)
│   │   ├── app-certmanager.yaml -> an app referencing this repo, but the 'cert-manager' folder (kustomize resources)
|   |   ├── app-gitlab-runner.yaml -> an app referencing this repo, but the 'gitlab-runner' folder (helm chart)
|   |   ├── app-projects.yaml -> an app referencing this repo, but the 'project-applications' folder (plain kubernetes resources)
│   │   ├── ...
│   │   ├── kustomization.yaml
│   ├── argocd-cm.yaml
│   ├── argocd-default-secret-delete.yaml
│   ├── argocd-ingress.yaml
│   ├── argocd-rbac-cm.yaml
│   ├── argocd-secret.yaml
│   ├── kustomization.yaml
│   └── namespace.yaml
├── cert-manager -> standard kustomize resources
│   ├── cert-manager-cluster-roles.yml
│   └── kustomization.yaml
├── gitlab-runner -> helm resources, in the templates folder are some custom resources, in the Chart.yaml is the reference to the gitlab-runner helmchart as a dependency
│   ├── templates
│   │   ├── minio-access-secret.yaml
│   │   ├── namespace.yaml
│   │   └── runner-token.yaml
│   ├── Chart.yaml
│   └── values.yaml
├── project-applications -> standard kubernetes resources (basically only additional apps referencing either this repo (all -settings app, or different git repositories, containing the actual content for the applications deployed by our teams))
│   ├── portals
│   │   ├── monitoring
│   │   │   ├── app-settings.yaml -> a settings app referenced in directly in this repo (see 'project-settings/portals/monitoring')
│   │   │   └── app.yaml -> a project app referencing a folder in a different repository (see next file tree below)
│   │   ├── project.yaml
│   │   └── repo-secret.yaml
│   └── helm-ikor-default-secret.yaml  -> the argocd secret containing the definition of the 'helm-ikor-default' helm repository (containing name,username,password,type,url)
├── project-settings
│   └── portals
│       └── monitoring
│           ├── templates
│           │   └── ...
│           ├── Chart.yaml
│           └── values.yaml
├── ... (more infra projects)
├── .gitlab-ci.yml
└── README.md

Project Gitops Repo

.
├── helm-generic-chart-test
│   ├── Chart.yaml
│   └── values.yaml
├──  ... (more "project" projects)
├── .gitignore
└── README.md

The Chart.yaml referencing the helm chart in the private repo looks like this:

apiVersion: v2
name: ikor-generic-chart
version: 1.0.2
dependencies:
  - name: generic-chart
    version: "1.0.0"
    # same url as configured in the helm repo secret
    repository: "https://our-repo-url"
    alias: test-service

The values.yaml for the charts looks like that (shortened):

test-service:
  deployment:
    image: nginx:1.21.6
    containerPort: 80

stieglma commented 2 years ago

@crenshaw-dev I tried a couple more things and this is what I found now:

when I execute the failed helm repo add command from the logs, directly in the commandline of the argocd-repo-server pod, there is no error, there is just a message that the repo already exists with identical settings, and so this command is skipped. Not sure where this difference to what happens according to the logs comes from
when I created an ArgoCD App which directly references the helm-chart (instead of in a subchart) everything works fine. The problem only occurs when the private repo is used as a subchart/dependency
I added sync-waves to make sure that the private helm-chart app is deployed as the very last app, to maybe work around race-conditions, but that also didn't help

next step will be to add all other helm-chart repos we are using as argocd-secrets and give them a name (I assume this will make them also appear outside the https:/ folder but instead in a folder with the name as foldername), to see if that changes anything

crenshaw-dev commented 2 years ago

when I execute the failed helm repo add command from the logs, directly in the commandline of the argocd-repo-server pod, there is no error, there is just a message that the repo already exists with identical settings, and so this command is skipped

What is the exit code of that command?

stieglma commented 2 years ago

the exit code is 0

crenshaw-dev commented 2 years ago

Yep, makes sense. I'm pretty heavily booked for a while, but I've added this to Thursday's contributor office hours doc. Maybe someone there will have an idea.

I think where we want to get is a minimal reproducible example. But isolating the relevant variables might require adding logging, dropping into a debugger, etc.

If no one's come up with anything by the end of the week, I'd be interested in hopping on a call next week.

stieglma commented 2 years ago

I did now create argocd repository secrets for all helm repositories we are using (I just omitted username/password where they are not necessary), and then I assigned this repos to the necessary ArgoCD projects.

So far everything seems to be working fine with that change (it was about 6 hours ago, the errors usually appeared much faster). I'll keep an eye on it the next few days.

If this actually resolved the problem then I'm sort of fine with it, but it would of course still be much nicer if I didn't need to create an argocd repository secret if it is not actually necessary due to passwords/usernames.

unfortunately I won't have time to also attend that meeting, I have something in parallel

crenshaw-dev commented 2 years ago

Interesting... I have no idea why that would work. What led you to try that?

stieglma commented 2 years ago

I thought it might be an interference between "implicit" helm repositories which are just used, but never declared in argocd as a helm repo directly (only as depencies in the Chart.yaml), and the explictly declared helm repo for our private/authorized chart server.

With that change I also got rid of that strange https: folder, because now for every helm repo there is a proper name, and not only the inferred one from the url

Up to now, still everything looking fine

crenshaw-dev commented 2 years ago

That is fascinating. And you still don't have any Apps pointing directly at those Helm repos?

stieglma commented 2 years ago

no, only transitively via dependencies

crenshaw-dev commented 2 years ago

Wow. Excellent find... I wonder if there's an initial fetch when the repo is added (to validate that the contents contain a chart maybe?) which caches the dependency in a way that allows the repo add calls to pass.

At any rate, now we have a workaround and a pretty good hint how to reproduce the issue.

crenshaw-dev commented 2 years ago

Assigning Alex because you has a pretty good idea what was going on here, but I'll also keep this issue in mind. Can't prioritize reproducing it just yet but hopefully soon.

stieglma commented 2 years ago

short update, the workaround still works fine, I had no further issues since I created helm-repository secrets for all non-private helm repos as well

ikogan commented 2 years ago

@stieglma could you elaborate on this workaround? I'm seeing this too when deploying any app that uses one of our charts as a subchart. We do have that repo added as a repo to argo and have several projects using it, here's some specifics:

Example Chart.yaml:

apiVersion: v2
name: mw-example-app
type: application
version: 2.0.0
maintainers:
dependencies:
    - name: common
      version: ">=0.0.2"
      repository: https://our.internal.artifactory.instance/artifactory/helm

Here's the repository's secret (managedFields and similar removed):

apiVersion: v1
data:
  name: artifactory
  password: redacted
  type: helm
  url: https://our.internal.artifactory.instance/artifactory/helm
  username: redacted
kind: Secret
metadata:
  annotations:
    managed-by: argocd.argoproj.io
  labels:
    argocd.argoproj.io/secret-type: repository
  name: repo-872087197
  namespace: argocd
type: Opaque

We also have a set of repositories configured in argocd-cm, but these are all public:

- name: bitnami
  type: helm
  url: https://charts.bitnami.com/bitnami
- name: jupyterhub
  type: helm
  url: https://jupyterhub.github.io/helm-chart/
- name: influxdata
  type: helm
  url: https://helm.influxdata.com/
- name: grafana
  type: helm
  url: https://grafana.github.io/helm-charts

repository.credentials includes our GitHub repo template only. We have a few projects but I'm only trying to use one with this helm subchart right now (some fields omitted for brevity):

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: mw-devtest
  namespace: argocd
spec:
  clusterResourceWhitelist:
    ...
  description: Middleware (Development/Test)
  destinations:
    ...
  orphanedResources:
    warn: false
  roles:
  - jwtTokens:
    ...
  sourceRepos:
  - https://our.internal.github/*
  - git@our.internal.github/*
  -  '*'

I'm experiencing two problems. If, on my application, I do not enable passCredentials, then our artifactory repo is added with no credentials, and we can't find the common chart. If I do enable passCredentials, then everything works directly after a bounce of argocd-repo-server. Right after it starts the /helm-working-dir directory in the pod is totally empty, even after syncing our impacted application.

Sometime after that, this directory fills up with repositories (including the fun https: one) and then I start getting the error in this issue.

I'm on ArgoCD 2.3.3, what am I missing from the workaround?

Edit: Is it possible that there's one app with passCredentials: false that causes a helm repo add with no credentials which then later ruins it for all other apps that have passCredentials: true?

stieglma commented 2 years ago

I don't have any specific settings done for passCredentials, but I created all helm repositories as separate secrets with the label argocd.argoproj.io/secret-type: repository even thought they don't need any credentials (you can just leave those empty). In the argocd-cm configmap I do not reference a single repository, I only do that on the AppProject resources

mmerickel commented 2 years ago

I was getting this error with passCredentials: true when I didn't need it. I set it to false and refresh starting working.

henryzhao95 commented 2 years ago

I am getting this error with passCredentials: true (and I need it). I've got an app-of-apps containing Foo API where passCredentials is set, and the subchart links to the external Helm repo which won't authenticate

ChrisMcKenzie commented 11 months ago

any update on this? going to try the workaround above but would be nice if I didn't have to.

shenli-awx commented 5 months ago

still happened in argo-cd-5.4.5

argoproj / argo-cd