Multiple Source 'ref' is not working

[ohjongsung]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

Multiple Source 'ref' is not working

error log :

Error: open <path to cached source>/argocd/guestbook/values.yaml: no such file or directory

To Reproduce

apply this command:

cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: helm-guestbook
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  annotations:
    argocd.argoproj.io/sync-wave: "-5"
spec:
  project: default

  sources:
    - repoURL: https://github.com/argoproj/argocd-example-apps.git
      targetRevision: master
      path: helm-guestbook
      helm:
        valueFiles:
          - $values/argocd/guestbook/values.yaml
    - repoURL: https://bitbucket.org/vktest12/mbbdo.git
      targetRevision: master
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: guestbook4

  syncPolicy:
    automated:
      prune: true
    syncOptions:
      - CreateNamespace=true
      - PrunePropagationPolicy=foreground
      - PruneLast=true
EOF

Expected behavior

find the value file exactly.

Screenshots

Version

argocd: v2.8.3+77556d9
  BuildDate: 2023-09-07T16:05:43Z
  GitCommit: 77556d9e64304c27c718bb0794676713628e435e
  GitTreeState: clean
  GoVersion: go1.20.6
  Compiler: gc
  Platform: linux/arm64

Logs

Failed to load target state: failed to generate manifest for source 1 of 2: rpc error: code = Unknown desc = `helm template . --name-template helm-guestbook --namespace guestbook4 --kube-version 1.27 --values <path to cached source>/argocd/guestbook/values.yaml --api-versions admissionregistration.k8s.io/v1 --api-versions admissionregistration.k8s.io/v1/MutatingWebhookConfiguration --api-versions admissionregistration.k8s.io/v1/ValidatingWebhookConfiguration --api-versions apiextensions.k8s.io/v1 --api-versions apiextensions.k8s.io/v1/CustomResourceDefinition --api-versions apiregistration.k8s.io/v1 --api-versions apiregistration.k8s.io/v1/APIService --api-versions apps/v1 --api-versions apps/v1/ControllerRevision --api-versions apps/v1/DaemonSet --api-versions apps/v1/Deployment --api-versions apps/v1/ReplicaSet --api-versions apps/v1/StatefulSet --api-versions argoproj.io/v1alpha1 --api-versions argoproj.io/v1alpha1/AppProject --api-versions argoproj.io/v1alpha1/Application --api-versions argoproj.io/v1alpha1/ApplicationSet --api-versions autoscaling/v1 --api-versions autoscaling/v1/HorizontalPodAutoscaler --api-versions autoscaling/v2 --api-versions autoscaling/v2/HorizontalPodAutoscaler --api-versions batch/v1 --api-versions batch/v1/CronJob --api-versions batch/v1/Job --api-versions certificates.k8s.io/v1 --api-versions certificates.k8s.io/v1/CertificateSigningRequest --api-versions coordination.k8s.io/v1 --api-versions coordination.k8s.io/v1/Lease --api-versions discovery.k8s.io/v1 --api-versions discovery.k8s.io/v1/EndpointSlice --api-versions events.k8s.io/v1 --api-versions events.k8s.io/v1/Event --api-versions flowcontrol.apiserver.k8s.io/v1beta2 --api-versions flowcontrol.apiserver.k8s.io/v1beta2/FlowSchema --api-versions flowcontrol.apiserver.k8s.io/v1beta2/PriorityLevelConfiguration --api-versions flowcontrol.apiserver.k8s.io/v1beta3 --api-versions flowcontrol.apiserver.k8s.io/v1beta3/FlowSchema --api-versions flowcontrol.apiserver.k8s.io/v1beta3/PriorityLevelConfiguration --api-versions helm.cattle.io/v1 --api-versions helm.cattle.io/v1/HelmChart --api-versions helm.cattle.io/v1/HelmChartConfig --api-versions k3s.cattle.io/v1 --api-versions k3s.cattle.io/v1/Addon --api-versions networking.k8s.io/v1 --api-versions networking.k8s.io/v1/Ingress --api-versions networking.k8s.io/v1/IngressClass --api-versions networking.k8s.io/v1/NetworkPolicy --api-versions node.k8s.io/v1 --api-versions node.k8s.io/v1/RuntimeClass --api-versions policy/v1 --api-versions policy/v1/PodDisruptionBudget --api-versions rbac.authorization.k8s.io/v1 --api-versions rbac.authorization.k8s.io/v1/ClusterRole --api-versions rbac.authorization.k8s.io/v1/ClusterRoleBinding --api-versions rbac.authorization.k8s.io/v1/Role --api-versions rbac.authorization.k8s.io/v1/RoleBinding --api-versions scheduling.k8s.io/v1 --api-versions scheduling.k8s.io/v1/PriorityClass --api-versions storage.k8s.io/v1 --api-versions storage.k8s.io/v1/CSIDriver --api-versions storage.k8s.io/v1/CSINode --api-versions storage.k8s.io/v1/CSIStorageCapacity --api-versions storage.k8s.io/v1/StorageClass --api-versions storage.k8s.io/v1/VolumeAttachment --api-versions v1 --api-versions v1/ConfigMap --api-versions v1/Endpoints --api-versions v1/Event --api-versions v1/LimitRange --api-versions v1/Namespace --api-versions v1/Node --api-versions v1/PersistentVolume --api-versions v1/PersistentVolumeClaim --api-versions v1/Pod --api-versions v1/PodTemplate --api-versions v1/ReplicationController --api-versions v1/ResourceQuota --api-versions v1/Secret --api-versions v1/Service --api-versions v1/ServiceAccount --include-crds` failed exit status 1: Error: open <path to cached source>/argocd/guestbook/values.yaml: no such file or directory

[bagel-dawg]

I'm seeing the same behaviour on 2.6 using this manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: app
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: namespace
  project: default
  sources:
  - repoURL: quay.io/oci-helmchart-repo
    chart: "chart"
    targetRevision: "0.0.2"
    helm:
      valueFiles:
      - $values/helm/values.yaml
      - $values/helm/sizing/small.yaml
  - repoURL: ssh://github.com:repo/org
    targetRevision: inf-4134
    ref: values
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true

[crenshaw-dev]

Can someone put together a public example to reproduce the issue? https://github.com/argoproj/argocd-example-apps/tree/master/helm-guestbook would probably be a good starting point.

[bagel-dawg]

@crenshaw-dev

This minimum code snippet with argocd-repo-server and argocd-application-set-controller for v2.6.15 fails for deploy 100% of the time:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-application
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: test-application
  project: default
  sources:
   - repoURL: 'https://helm.github.io/examples'
     chart: hello-world
     targetRevision: 0.1.0
     helm:
       valueFiles:
       - $values/charts/prometheus/values.yaml
   - repoURL: 'https://github.com/prometheus-community/helm-charts'
     targetRevision: main
     ref: values
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true

CLI output when trying to sync manually:

$ argocd --core app sync test-applcation
FATA[0001] rpc error: code = FailedPrecondition desc = rpc error: code = Unknown desc = repository "" cannot be initialized: repo URL is invalid

CLI output when doing a "get":

$ argocd --core app get test-application
Name:               test-application
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          test-application
URL:                http://localhost:38029/applications/test-application
Repo:
Target:
Path:
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Unknown
Health Status:      Healthy

CONDITION        MESSAGE  LAST TRANSITION
ComparisonError  rpc error: code = Unknown desc = `helm template . --name-template turbine --namespace swimlane --kube-version 1.24 --values /tmp/9dcba7f0-33e5-4ba6-9554-e69bcb533c35/hello-world/charts/prometheus/values.yaml [...] --include-crds` failed exit status 1: Error: open /tmp/9dcba7f0-33e5-4ba6-9554-e69bcb533c35/hello-world/charts/prometheus/values.yaml: no such file or directory  2023-09-25 16:13:50 -0400 EDT

[crenshaw-dev]

Hm. I'm unable to reproduce the issue in either 2.6.15 or master.

[crenshaw-dev]

Ah, reproduced it.

$ kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.5.0/manifests/crds/application-crd.yaml
customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io configured
$ argocd app sync test-application
FATA[0000] rpc error: code = FailedPrecondition desc = error resolving repo revision: rpc error: code = Unknown desc = repository "" cannot be initialized: repo URL is invalid 

Looks like you're using a version of the Application CRD which predates the multi-source feature. Apply the Application CRD which corresponds to the Argo CD version you're running.

[bagel-dawg]

I think the repo URL is invalid could be related to a version difference between my local argocd CLI (2.3.3) and the applied server version. (2.6.15/2.8.4)

However, even on 2.8.4 I am still having issues with the example provided verbatim. I will try next on a new install that has never been used before to exclude any previously-installed-CRD version issues.

[bagel-dawg]

This is incredibly weird - I am able to reproduce this consistently even using the "stable" deployment method, makes me think it must be an issue with the spec but I have no idea where to start.

To reliably reproduce on stable:

# Brand new KinD cluster, eks v1.27
$ kind create cluster
[...]

# Install ArgoCD as suggested by install guide:
$ kubectl create namespace argocd
$ kubectl apply  --namespace argocd -f  https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
[......]

# Capture versions before continuing
$ argocd --core version
argocd: v2.8.4+c279299
  BuildDate: 2023-09-13T19:43:37Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.7
  Compiler: gc
  Platform: linux/amd64
argocd-server: v2.8.4+c279299
  BuildDate: 2023-09-13T19:43:37Z
  GitCommit: c27929928104dc37b937764baf65f38b78930e59
  GitTreeState: clean
  GoVersion: go1.20.7
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v4.5.2 2022-02-09T23:26:42Z
  Helm Version: v3.8.0+gd141386
  Kubectl Version: v0.24.2
  Jsonnet Version: v0.20.0

# Apply test case application
$ cat <<EOF | kubectl apply -f -
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-application
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: test-application
  project: default
  sources:
   - repoURL: 'https://helm.github.io/examples'
     chart: hello-world
     targetRevision: 0.1.0
     helm:
       valueFiles:
       - $values/charts/prometheus/values.yaml
   - repoURL: 'https://github.com/prometheus-community/helm-charts'
     targetRevision: main
     ref: values
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true
EOF

# Check application status
$ argocd --core app get test-application
Name:               argocd/test-application
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          test-application
URL:                http://localhost:46685/applications/test-application
Repo:               https://helm.github.io/examples
Target:             0.1.0
Path:
Helm Values:        /charts/prometheus/values.yaml
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Unknown
Health Status:      Healthy

CONDITION        MESSAGE  LAST TRANSITION
ComparisonError  Failed to load target state: failed to generate manifest for source 1 of 2: rpc error: code = Unknown desc = Manifest generation error (cached): `helm template . --name-template test-application --namespace test-application --kube-version 1.27 --values /tmp/5af1bf12-4706-4730-9ec2-3e02419f9ee1/hello-world/charts/prometheus/values.yaml [.....] --include-crds` failed exit status 1: Error: open /tmp/5af1bf12-4706-4730-9ec2-3e02419f9ee1/hello-world/charts/prometheus/values.yaml: no such file or directory  2023-09-25 17:18:03 -0400 EDT

[crenshaw-dev]

That's a different error message from what you were getting previously. The new error message is because the charts/prometheus directory doesn't exist in the specified repo.

[bagel-dawg]

@crenshaw-dev It was the same error message that I posted in this comment previously when doing a get. This is also the same error as the original post.

CLI output when doing a "get":

$ argocd --core app get test-application
Name:               test-application
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          test-application
URL:                http://localhost:38029/applications/test-application
Repo:
Target:
Path:
SyncWindow:         Sync Allowed
Sync Policy:        Automated (Prune)
Sync Status:        Unknown
Health Status:      Healthy

CONDITION        MESSAGE  LAST TRANSITION
ComparisonError  rpc error: code = Unknown desc = `helm template . --name-template turbine --namespace swimlane --kube-version 1.24 --values /tmp/9dcba7f0-33e5-4ba6-9554-e69bcb533c35/hello-world/charts/prometheus/values.yaml [...] --include-crds` failed exit status 1: Error: open /tmp/9dcba7f0-33e5-4ba6-9554-e69bcb533c35/hello-world/charts/prometheus/values.yaml: no such file or directory  2023-09-25 16:13:50 -0400 EDT```

Unless I'm misreading your earlier reply, the manifest you tested with is the same as the one that produced this error and the one on KinD.

The values file also does exist in the remote values repo here

[anandf]

@bagel-dawg Could it be possible that the $values is treated as a shell variable as its applied using a cat shell command ?

Can you try escaping the $ char by prepending it with \ something like below

       valueFiles:
       - "\$values/charts/prometheus/values.yaml"

By escaping, it will no longer be treated as a shell variable.

If treated as a shell variable, the path becomes /charts/prometheus/values.yaml instead of $values/charts/prometheus/values.yaml

I wrote a small script to verify this behaviour with/without escaping and confirmed this behaviour.

cat  /tmp/test.sh

Output:

cat <<EOF | tee /tmp/test.log 
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test-application
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    server: https://kubernetes.default.svc
    namespace: test-application
  project: default
  sources:
   - repoURL: 'https://helm.github.io/examples'
     chart: hello-world
     targetRevision: 0.1.0
     helm:
       valueFiles:
       - \$values/charts/prometheus/values.yaml
   - repoURL: 'https://github.com/prometheus-community/helm-charts'
     targetRevision: main
     ref: values
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true
EOF

[agardnerIT]

I hit this too on argo v2.8.4+c279299

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "1"
  labels:
    dt.owner: "platform_team"
  name: opentelemetry-collector
  namespace: argocd
spec:
  sources:
    - repoURL: 'https://open-telemetry.github.io/opentelemetry-helm-charts'
      targetRevision: 0.71.0
      chart: opentelemetry-collector
      # helm:
      #   values: |
      #     mode: "daemonset"
      helm:
        valueFiles:
        - \$values/gitops/manifests/opentelemetry/values.yaml
    - repoURL: 'https://github.com/agardnerit/test123.git'
      ref: values
  destination:
    namespace: opentelemetry
    server: 'https://kubernetes.default.svc'
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    retry:
      limit: 5
      backoff:
        duration: 5s
        maxDuration: 3m0s
        factor: 2
    syncOptions:
      - CreateNamespace=true

[cmoulliard]

I dont see how we should or could pass correctly the helmValues as the following config dot not apply properly my helm values

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: janus-idp
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true
  project: default
  destination:
    # cluster API URL
    server: https://kubernetes.default.svc
    namespace: idp
  sources:
    - repoURL: 'https://janus-idp.github.io/helm-backstage'
      chart: backstage
      targetRevision: 2.6.0
      helm:
        valueFiles:
        - '$values/local/helm-values/janus-idp.yml'
    - repoURL: 'https://github.com/ch007m/my-janus.git'
      targetRevision: main
      ref: values
    - repoURL: 'https://github.com/ch007m/my-janus.git'
      targetRevision: main
      path: charts/janus-idp

[bagel-dawg]

@anandf You were spot-on with the variable substitution from my cat-EOF combo. Apply directly as a file instead of wrapped in an EOF works just as expected.

[cmoulliard]

You were spot-on with the variable substitution from my cat-EOF combo. Apply directly as a file instead of wrapped in an EOF works just as expected.

Correct. That works for me including also passing as HTTP url the reference of the helmValues

        valueFiles:
        # That works: We can pass an URL pointing to the values file directly
        #- https://raw.githubusercontent.com/ch007m/my-janus/main/local/values/janus-idp.yml
        # or  
        # as alternative use the $values resolving mechanism
        - $values/local/values/janus-idp.yml
    - repoURL: 'https://github.com/ch007m/my-janus.git'
      targetRevision: main
      ref: values

[ishitasequeira]

Closing the issue with @anandf 's comment as the solution. Feel free to open the issue in case you are still facing it.