When deploying multiple versions of a Helm Chart through Kustomize only the first defined version is used.

[unknownbanana]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

When using Kustomize and defining multiple version of the same chart in a single kustomization.yml file Argo will always use the first defined version it finds in the list of Helm Chart definitions.

Due to Argo using the first version it finds in the list of charts/releases a syncing issue occurs. The syncing issue is caused by the older versions of the chart not defining the new required values.

To Reproduce

Context:

• We use private Helm Charts.
• We use an OCI private registry.
• We work in an airgapped environment.

In short, I cannot provide a true reproduction since the charts are not public and, I do not have access to public charts. I can however provide an example to illustrate the issue.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../a-bunch/of/resources.yaml

helmCharts:
  - name: helm-chart-name
    repo: oci://private.registry.example.org/project-name
    version: 1.0
    releaseName: release-name-0
    valuesInline:
      a_bunch:
        of: values
        that: "do not exist in the older version"
        which:
          causes: a
          lot: of problems

  - name: helm-chart-name
    repo: oci://private.registry.example.org/project-name
    version: 0.3
    releaseName: release-name-1
    valuesInline:
      a_bunch:
        of: values

  - name: helm-chart-name
    repo: oci://private.registry.example.org/project-name
    version: 0.3
    releaseName: release-name-2
    valuesInline:
      a_bunch:
        of: values

  - name: helm-chart-name
    repo: oci://private.registry.example.org/project-name
    version: 0.3
    releaseName: release-name-3
    valuesInline:
      a_bunch:
        of: values

  - name: helm-chart-name
    repo: oci://private.registry.example.org/project-name
    version: 0.3
    releaseName: release-name-4
    valuesInline:
      a_bunch:
        of: values

Expected behavior

The expected behavior is as follows, ArgoCD:

• Creates resources based on the defined version for each release.
• Does not re-use a defined version for each defined release.

Screenshots

Version

2.10.3

Logs

one or more objects failed to apply, reason: error when patching "/dev/shm/3511515270": admission webhook "webhook.pipeline.tekton.dev" denied the request: mutation failed: cannot create patch for round tripped newBytes: cannot marshal interface: json: error calling MarshalJSON for type v1beta1.ParamValue: impossible ParamValues.Type: "". Retrying attempt #1 at 7:58AM.

[tdorianh]

I can replicate this using the following publicly available helm charts, on Argo CD server version v2.11.3:

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
  - releaseName: istiod-blue
    name: istiod
    repo: https://istio-release.storage.googleapis.com/charts
    version: 1.22.1
    namespace: istio-system
    valuesInline:
      revision: 1-22-1
      revisionTags:
        - blue

  - releaseName: istiod-green
    name: istiod
    repo: https://istio-release.storage.googleapis.com/charts
    version: 1.23.2
    namespace: istio-system
    valuesInline:
      revision: 1-23-2
      revisionTags:
        - green

This results in 2 Istio deployments both running 1.22.1 instead of one running 1.22.1 and one running 1.23.2.

Running argocd app diff manually shows that the CLI recognizes that the correct result is the image 1.22.1 for one deployment and 1.23.2 for the other deployment. (that is, argocd CLI disagrees with argocd server, both running v2.11.3+3f344d5)

However, Argo CD used 1.22.1 for both, and does not show anything out of sync.

This bug complicates testing upgrades of helm charts side-by-side using Argo CD.

[andrii-korotkov-verkada]

ArgoCD versions 2.10 and below have reached EOL. Can you upgrade and tell us if the issue is still present, please?

[unknownbanana]

ArgoCD versions 2.10 and below have reached EOL. Can you upgrade and tell us if the issue is still present, please?

Hi @andrii-korotkov-verkada , unfortunately I cannot. We're running on OpenShift, GitOps Operator (see: compatibility-matrix) and RedHat has not released a version of that Operator which supports ArgoCD 2.12.x.

As soon as they support 2.12..x I can check.

[andrii-korotkov-verkada]

What about the most recent 2.11?

[unknownbanana]

What about the most recent 2.11?

I think @tdorianh ran my use-case for 2.11.3. which failed. I can check if the issue is still present for 2.12.3. I should note however that I'm limited in my ability to test specific versions as I'm bound to the version the OpenShift GitOps operator is using. If you have a specific ArgoCD version in mind then running @tdorianh 's example yourself would probably be quicker than me trying to get a version running locally.

[unknownbanana]

@andrii-korotkov-verkada I just verified that this is resolved for 2.12.3. I'm able to deploy 2 different versions. Would still like to see someone that's using public charts reproduce that result however. Since the use-cases are similar but not exactly equivalent.

[reggie-k]

@tdorianh could you check whether still reproducible with Argo CD 2.13 version for public charts?