Unable to upgrade Argo CD to any newer version then 2.5.2 (Error: no cached repository for helm-manager)

[jacek-jablonski]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

I cannot update ArgoCD to any version above 2.5.2. Version 2.5.2 is the last version of ArgoCD that works fine for me. When I update, for example, to version 2.5.5 and try to refresh the application, an error appears and the application has the Unknown status. After a while, some applications (253) also go to the Unknown status. The other applications (75) do not change and the reconciliation works correctly. I've tried all the methods I know of, such as cleaning redis, restarting argocd-repo-server, unfortunately nothing helps. Rolling back to version 2.5.2 makes everything work properly again. The only common pattern I observe for these 253 applications in the Unknown status is that they use a private Helm chart repo and use one chart from this repo as a subchart. But this is not the case for all applications. Some applications using a similar scheme work properly. I do not commit Chart.lock files or the templates directory in my applications.

To Reproduce

1. Update ArgoCD to version 2.5.3 or 2.5.4 or 2.5.5.
2. Refresh any application that uses a chart placed on a private repo of Helm charts.
3. After refreshing the application has Unknown status.
4. After some time, some applications also have the Unknown status.

Expected behavior

After updating ArgoCD to version 2.5.5 everything works fine.

Version

╰─❯ argocd version
argocd: v2.5.4+86b2dde.dirty
  BuildDate: 2022-12-07T01:46:20Z
  GitCommit: 86b2dde8e4bf1187acd2b4294e94451cd104dad8
  GitTreeState: dirty
  GoVersion: go1.19.4
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v2.5.2+148d8da
  BuildDate: 2022-11-07T16:42:47Z
  GitCommit: 148d8da7a996f6c9f4d102fdd8e688c2ff3fd8c7
  GitTreeState: clean
  GoVersion: go1.18.8
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.7 2022-08-02T16:35:54Z
  Helm Version: v3.10.1+g9f88ccb
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.18.0

Logs

A lot of logs like:

time="2022-12-17T13:49:28Z" level=error msg="finished unary call with code Unknown" error="`helm dependency build` failed exit status 1: Error: no cached repository for helm-manager-be3ed640d7b4c751119456f6bcc217ce1ef34bd7b2a63077acbc7c305d80945e found. (try 'helm repo update'): open /helm-working-dir/repository/helm-manager-be3ed640d7b4c751119456f6bcc217ce1ef34bd7b2a63077acbc7c305d80945e-index.yaml: no such file or directory" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2022-12-17T13:49:27Z" grpc.time_ms=903.045 span.kind=server system=grpc

[jacek-jablonski]

I've narrowed down the problem to this commit: https://github.com/argoproj/argo-cd/commit/0366e0153 Dropping this commit and rebuilding Argo CD causes Argo working without issues.

[jemag]

I am having the same problem. In my case I am using kustomized-helm configmap plugin, and applications stuck in unknown status use a private helm chart repos. It was working fine prior to the upgrade from 2.5.1 to 2.6.0.

Now getting logs such as:

time="2023-02-07T20:06:42Z" level=info msg="/bin/sh -c helm dependency build || true" dir=/tmp/_argocd-repo/7f9abf7e-b2db-46e6-92fa-a70f00059156/k8s/spine/base execID=b3323
time="2023-02-07T20:06:43Z" level=info msg=Trace args="[/bin/sh -c helm dependency build || true]" dir=/tmp/_argocd-repo/7f9abf7e-b2db-46e6-92fa-a70f00059156/k8s/spine/base operation_name="exec /bin/sh" time_ms=85.958799
time="2023-02-07T20:06:43Z" level=error msg="`/bin/sh -c helm template . --name-template $ARGOCD_APP_NAME --namespace $ARGOCD_APP_NAMESPACE --include-crds -f $ARGOCD_ENV_VALUES_PATH > all.yaml && kustomize build $ARGOCD_ENV_OVERLAY_PATH` failed exit status 1: Error: An error occurred while checking for chart dependencies. You may need to run `helm dependency build` to fetch missing dependencies: found in Chart.yaml, but missing in charts/ directory: bootstrap-team-chart" execID=afc29
time="2023-02-07T20:06:43Z" level=error msg="finished unary call with code Unknown" error="`/bin/sh -c helm template . --name-template $ARGOCD_APP_NAME --namespace $ARGOCD_APP_NAMESPACE --include-crds -f $ARGOCD_ENV_VALUES_PATH > all.yaml && kustomize build $ARGOCD_ENV_OVERLAY_PATH` failed exit status 1: Error: An error occurred while checking for chart dependencies. You may need to run `helm dependency build` to fetch missing dependencies: found in Chart.yaml, but missing in charts/ directory: bootstrap-team-chart" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2023-02-07T20:06:42Z" grpc.time_ms=565.063 span.kind=server system=grpc

Here the dependency error is hidden because of helm dependency build || true, but if I temporarily remove it we get same kind of error as yours:

This is unfortunate since I was really looking forward to using some of the new features from 2.6.0

[tyrken]

FYI just looking to upgrade from 2.5.2 also, and while reading the changelog I came across https://github.com/argoproj/argo-cd/pull/11285 which is in 2.5.3 ... which worries me given we use helm & helm plugins pre-installed on a custom image & I remember previous problems in this area.

I've not tried to upgrade yet so can't say we're broken but that change disagreeing with https://github.com/argoproj/argo-cd/blob/8e066483270acb19bb81895c1c372ee055cd0904/manifests/base/repo-server/argocd-repo-server-deployment.yaml#L146-L151 makes me a bit more nervous...

[lacebal]

We are using Helm v3 with S3 repository plugin and I have narrow down the problem to the commit pointed by @jacek-jablonski https://github.com/argoproj/argo-cd/commit/0366e0153

That commit renders this documentation outdated https://argo-cd.readthedocs.io/en/stable/user-guide/helm/#helm-plugins and bakres any Helm Chart using a private S3 repository that has being added through an initContainer.

Changing HELM_CONFIG_HOME to a temporal one makes any pre-added repository to not be visible to Helm. This together with the very limited code to add Helm repositories on demand that only works with HTTPS and OCI schemas, makes this a breaking change.