Failed to generate manifest for source (helm dependency build)

[volver-13]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

We have recently migrated CMP to sidecars which is a pre-requiste when upgrading from v2.7 to v2.8 as per the instructions.

We use ArgoCD Vault Plugin and we have argocd-repo-server running with 5 sidecars now:

• argocd-vault-plugin
• argocd-vault-plugin-helm
• argocd-vault-plugin-helm-with-args
• argocd-vault-plugin-helm-with-args-custom-release
• argocd-vault-plugin-helm-with-values

We mostly rely on argocd-vault-plugin-helm-with-args plugin for our applications but right now after upgrade we noticed the following errors:

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: `sh -c "helm dependency build"` failed exit status 1: Error: no repository definition for https://prometheus-community.github.io/helm-charts. Please add the missing repos via 'helm repo add'

We used to see these errors before for the charts deployment that had chart dependencies specified in Chart.yaml like for example (robusta helm chart):

apiVersion: v2
appVersion: 0.10.21
dependencies:
- condition: enablePrometheusStack
  name: kube-prometheus-stack
  repository: https://prometheus-community.github.io/helm-charts
  version: 47.2.0
description: Robusta Helm chart for Kubernetes
name: robusta
type: application
version: 0.10.21

We tried adding helm charts to ArgoCD repos but it never solved the issue.

So far our workaround was to kubectl exec into argocd-repo-server container and execute helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

When ArgoCD services are restarted the issue reoccurs. It would be really annoying especially now considering the fact of having 5 sidecars and couple of repos to be added.

To Reproduce

Create ApplicationSet as follows:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: robusta
spec:
  generators:
  - list:
      elements:
      - cluster_name: STAGING_2
        cluster: staging-2
        app_version: 0.10.21
  template:
    metadata:
      name: '{{cluster}}-robusta'
    spec:
      project: platform
      source:
        path: resources/robusta/helm-chart
        repoURL: https://gitlab.com/argo/platform.git
        targetRevision: 'robusta-{{app_version}}'
        plugin:
          name: argocd-vault-plugin-helm-with-args
          env:
            - name: HELM_ARGS
              value: "-f values.yaml -f cluster-values/{{cluster}}-values.yaml --namespace robusta"
      destination:
        name: '{{cluster_name}}'
        namespace: robusta
      syncPolicy:
        syncOptions:
        - ApplyOutOfSyncOnly=true
        - CreateNamespace=true

The following repo https://gitlab.com/argo/platform/resources/robusta/helm-chart contains robusta helm chart (helm pull robusta/robusta --version 0.10.21)

Expected behavior

Chart is templated correctly with vault secrets.

Screenshots

ArgoCD helm chart: argo/argo-cd, version 5.43.4
v2.8.0+804d4b8

Logs

argocd-application-controller-0 application-controller time="2023-08-23T12:39:42Z" level=info msg="Normalized app spec: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2023-08-23T12:39:42Z\",\"message\":\"Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): plugin sidecar failed. error generating manifests in cmp: rpc error: code = Unknown desc = error generating manifests: `sh -c \\\"helm dependency build\\\"` failed exit status 1: Error: no repository definition for https://prometheus-community.github.io/helm-charts. Please add the missing repos via 'helm repo add'\",\"type\":\"ComparisonError\"}]}}" application=argocd/prod-robusta

[zswanson]

Have you tried adding the helm repo update to the init phase of the plugin? https://argo-cd.readthedocs.io/en/stable/operator-manual/config-management-plugins/#write-the-plugin-configuration-file

[damoshushu]

argocd synchronizing does not respect to the syncOption "CreateNamespace=true". It tried to do helm template before creating the namespace, so it failed because there is no namespace

[josephbarnett]

For others that see this error. I received this same error on a different chart. After some digging, I found the chart index file had been corrupted. Specifically there as a published chart (tgz) but no record in the index.yaml file.

I can't say this is the same issue you are experiencing, but I recommend check the published index.yaml and chart tgz file exists.

[rjshrjndrn]

If you're using value from another repo, whose deletion trigger the application delete cause this error, as I'm using a custom plugin.