helm values file is unable to recognize the secret file which is encrypted using sops

[priyas16]

Current Behavior

We are trying to deploy our changes using helm in Argo CD. Our Application file has reference to both the values.yaml and the secret which is encrypted using sops like below. But its not getting deployed, saying the fields referenced through the encrypted file are not set.

helm: valueFiles:

• values.yaml
• secrets+age-import:///helm-secrets-private-keys/key.txt?accessToken.enc.yaml

Have verified manually that the encryption is working properly by decrypting the values and was able to deploy the helm manually with the same secret fields. Also in order to debug tried passing one of the field value directly inside the values.yaml and that error is not seen now. We would require your help to resolve this issue. Thanks!

Expected Behavior

We are trying to deploy our changes using helm in Argo CD. Our Application file has reference to both the values.yaml and the secret which is encrypted using sops like below.

helm: valueFiles:

• values.yaml
• secrets+age-import:///helm-secrets-private-keys/key.txt?accessToken.enc.yaml

We are expecting the values.yaml file recognises the accessToken.enc.yaml and deploy the changes.

Steps To Reproduce

No response

Environment

• Helm Version: 3.14.0
• Helm Secrets Version: 4.4.2
• ArgoCD Version : V2.10.3
• OS: RHEL 8.9
• Shell: 4.4

Anything else?

No response

[jkroepke]

tbh. it's not easy to debug from my side, but I feel one thing a bit strange:

But its not getting deployed, saying the fields referenced through the encrypted file are not set.

An age encrypted file looks like this:

https://github.com/jkroepke/helm-secrets/blob/e473cb3a53fafa0990581a00aba18532feb886c9/tests/assets/values/sops/secrets.age.yaml#L1-L4

Even the files are not decrypted by help, the keys should be still present, but the values should be encrypted only. Could you confirm that this:

helm:
  valueFiles:
  - values.yaml
  - accessToken.enc.yaml

is not return any errors?

[priyas16]

tbh. it's not easy to debug from my side, but I feel one thing a bit strange:

But its not getting deployed, saying the fields referenced through the encrypted file are not set.

An age encrypted file looks like this:

https://github.com/jkroepke/helm-secrets/blob/e473cb3a53fafa0990581a00aba18532feb886c9/tests/assets/values/sops/secrets.age.yaml#L1-L4

Even the files are not decrypted by help, the keys should be still present, but the values should be encrypted only. Could you confirm that this:

helm:
  valueFiles:
  - values.yaml
  - accessToken.enc.yaml

is not return any errors?

Hi jkroepke,

Yes, we are seeing the error. Have attached it for your reference.

[jkroepke]

If

helm:
  valueFiles:
  - values.yaml
  - accessToken.enc.yaml

returns the same error, then the keys accessToken.enc.yaml are wrong. The file should contain splunkObservability.realm or splunkPlatform.endpoint.

[priyas16]

Hi jkroepke,

Thanks for your response. Please find below the different error message for 2 scenarios of passing the encrypted file.

Scenario1:

helm: valueFiles:

• values.yaml
• secrets+age-import:///helm-secrets-private-keys/key.txt?accessToken.enc.yaml

Error:

Scenario2:

helm: valueFiles:

• values.yaml
• accessToken.enc.yaml

Error:

But the keys are correct we did verify by decrypting them.

Could you please check and let us know your thoughts. Thanks!

[jkroepke]

Could it be possible that you miss an additional value property which is required in addition?

Since, we can see a difference between encrypted an non-encrypted. I expected that encryption works as expected.

[ajaykumar4]

@jkroepke even I'm facing same issue I add all the configuration mentioned in document for argocd and sops with age. In argo application.yaml

helm:
  valueFiles:
    - values.yaml
    - values.sops.yaml

Getting error values.sops.yaml file not found.

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): `helm template . --name-template argocd --namespace argocd --kube-version 1.30 --values secrets://values.sops.yaml <api versions removed> --include-crds` failed exit status 1: [helm-secrets] File does not exist: values.sops.yaml Error: failed parsing --set-file data: plugin "scripts/run.sh downloader" exited with error [helm-secrets] helm template command errored on value 'secrets://values.sops.yaml' [helm-secrets] File does not exist: secrets://values.sops.yaml Error: plugin "secrets" exited with error

If I remove values.sops.yaml, then it works fine. https://github.com/ajaykumar4/home-lab-argocd/tree/main/kubernetes/apps/argocd/argocd

[Harishkrishna17]

@jkroepke Even we have the same problem , Pasting my configuration below

Please help

argocd helm chart file :

repoServer:
  env:
  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container
  - name: HELM_PLUGINS
    value: /custom-tools/helm-plugins/
  - name: HELM_SECRETS_SOPS_PATH
    value: /custom-tools/sops
  - name: HELM_SECRETS_KUBECTL_PATH
    value: /custom-tools/kubectl
  volumeMounts:
  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container
  - mountPath: /custom-tools
    name: custom-tools
  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#method-1-mount-the-private-key-from-a-kubernetes-secret-as-volume
  - mountPath: /helm-secrets-private-keys/
    name: helm-secrets-private-keys

  # -- Additional volumes to the repo server pod
  volumes:
  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container
  - name: custom-tools
    emptyDir: {}
  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#method-1-mount-the-private-key-from-a-kubernetes-secret-as-volume
  - name: helm-secrets-private-keys
    secret:
      secretName: helm-secrets-private-keys
  initContainers:
  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container
  - name: download-tools
    image: alpine:latest
    command: [sh, -ec]
    env:
    - name: HELM_SECRETS_VERSION
      value: "4.6.0"
    - name: KUBECTL_VERSION
      value: "1.30.1"
    - name: VALS_VERSION
      value: "0.37.1"
    - name: SOPS_VERSION
      value: "3.8.1"
    args:
    - |
      mkdir -p /custom-tools/helm-plugins
      wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf-;
      wget -qO /custom-tools/curl https://github.com/moparisthebest/static-curl/releases/latest/download/curl-amd64
      wget -qO /custom-tools/sops https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux.amd64
      wget -qO /custom-tools/kubectl https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl
      wget -qO- https://github.com/helmfile/vals/releases/download/v${VALS_VERSION}/vals_${VALS_VERSION}_linux_amd64.tar.gz | tar -xzf- -C /custom-tools/ vals;
      cp /custom-tools/helm-plugins/helm-secrets/scripts/wrapper/helm.sh /custom-tools/helm
      chmod +x /custom-tools/*
    volumeMounts:
    - mountPath: /custom-tools
      name: custom-tools

argo yaml file

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: harish
  namespace: argocd
spec:
  project: default
  source:
    repoURL: my-repo
    targetRevision: master
    path: helm-charts/platform/kube-prometheus-stack
    helm:
      valueFiles:
      - values-dev.yaml
      - secrets+age-import:///helm-secrets-private-keys/key.txt?values.sops.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

argocd confimap :

apiVersion: v1
data:
  admin.enabled: "true"
  application.instanceLabelKey: argocd.argoproj.io/instance
  exec.enabled: "false"
  helm.valuesFileSchemes: secrets+gpg-import, secrets+gpg-import-kubernetes, secrets+age-import,
    secrets+age-import-kubernetes, secrets,secrets+literal, https
  server.rbac.log.enforce.enable: "false"
  statusbadge.enabled: "false"
  timeout.hard.reconciliation: 0s
  timeout.reconciliation: 180s
  url: https://argocd.example.com
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: argocd
    meta.helm.sh/release-namespace: argocd
  creationTimestamp: "2024-07-05T14:30:12Z"
  labels:
    app.kubernetes.io/component: server
    app.kubernetes.io/instance: argocd
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
    app.kubernetes.io/version: v2.11.3
    helm.sh/chart: argo-cd-7.1.3
  name: argocd-cm
  namespace: argocd
  resourceVersion: "3889857"
  uid: ce560fb5-5800-4ca3-bde0-78293095eea7

error:

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = helm template . --name-template harish --namespace argocd --kube-version 1.29 --values <path to cached source>/helm-charts/platform/kube-prometheus-stack/values-dev.yaml --values secrets+age-import:///helm-secrets-private-keys/key.txt?values.sops.yaml <api versions removed> --include-crds failed exit status 1: No keys found in file [helm-secrets] Error while decrypting file: values.sops.yaml Error: plugin "scripts/run.sh downloader" exited with error

[jkroepke]

@jkroepke even I'm facing same issue

I add all the configuration mentioned in document for argocd and sops with age.

In argo application.yaml

helm:

  valueFiles:

    - values.yaml

    - values.sops.yaml

Getting error values.sops.yaml file not found.

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached): `helm template . --name-template argocd --namespace argocd --kube-version 1.30 --values secrets://values.sops.yaml <api versions removed> --include-crds` failed exit status 1: [helm-secrets] File does not exist: values.sops.yaml Error: failed parsing --set-file data: plugin "scripts/run.sh downloader" exited with error [helm-secrets] helm template command errored on value 'secrets://values.sops.yaml' [helm-secrets] File does not exist: secrets://values.sops.yaml Error: plugin "secrets" exited with error

If I remove values.sops.yaml, then it works fine.

https://github.com/ajaykumar4/home-lab-argocd/tree/main/kubernetes/apps/argocd/argocd

Last time I used ArgoCD, it does not extract the helm tar.gz package that contains the secrets.yaml. helm-secrets just looks on the file system and may not found the file on the filesystem.

[jkroepke]

@jkroepke Even we have the same problem , Pasting my configuration below

Please help

argocd helm chart file :

repoServer:

  env:

  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container

  - name: HELM_PLUGINS

    value: /custom-tools/helm-plugins/

  - name: HELM_SECRETS_SOPS_PATH

    value: /custom-tools/sops

  - name: HELM_SECRETS_KUBECTL_PATH

    value: /custom-tools/kubectl

  volumeMounts:

  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container

  - mountPath: /custom-tools

    name: custom-tools

  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#method-1-mount-the-private-key-from-a-kubernetes-secret-as-volume

  - mountPath: /helm-secrets-private-keys/

    name: helm-secrets-private-keys

  # -- Additional volumes to the repo server pod

  volumes:

  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container

  - name: custom-tools

    emptyDir: {}

  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#method-1-mount-the-private-key-from-a-kubernetes-secret-as-volume

  - name: helm-secrets-private-keys

    secret:

      secretName: helm-secrets-private-keys

  initContainers:

  ## See https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md#option-2-init-container

  - name: download-tools

    image: alpine:latest

    command: [sh, -ec]

    env:

    - name: HELM_SECRETS_VERSION

      value: "4.6.0"

    - name: KUBECTL_VERSION

      value: "1.30.1"

    - name: VALS_VERSION

      value: "0.37.1"

    - name: SOPS_VERSION

      value: "3.8.1"

    args:

    - |

      mkdir -p /custom-tools/helm-plugins

      wget -qO- https://github.com/jkroepke/helm-secrets/releases/download/v${HELM_SECRETS_VERSION}/helm-secrets.tar.gz | tar -C /custom-tools/helm-plugins -xzf-;

      wget -qO /custom-tools/curl https://github.com/moparisthebest/static-curl/releases/latest/download/curl-amd64

      wget -qO /custom-tools/sops https://github.com/getsops/sops/releases/download/v${SOPS_VERSION}/sops-v${SOPS_VERSION}.linux.amd64

      wget -qO /custom-tools/kubectl https://dl.k8s.io/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl

      wget -qO- https://github.com/helmfile/vals/releases/download/v${VALS_VERSION}/vals_${VALS_VERSION}_linux_amd64.tar.gz | tar -xzf- -C /custom-tools/ vals;

      cp /custom-tools/helm-plugins/helm-secrets/scripts/wrapper/helm.sh /custom-tools/helm

      chmod +x /custom-tools/*

    volumeMounts:

    - mountPath: /custom-tools

      name: custom-tools

argo yaml file

apiVersion: argoproj.io/v1alpha1

kind: Application

metadata:

  name: harish

  namespace: argocd

spec:

  project: default

  source:

    repoURL: my-repo

    targetRevision: master

    path: helm-charts/platform/kube-prometheus-stack

    helm:

      valueFiles:

      - values-dev.yaml

      - secrets+age-import:///helm-secrets-private-keys/key.txt?values.sops.yaml

  destination:

    server: https://kubernetes.default.svc

    namespace: argocd

  syncPolicy:

    automated:

      prune: true

      selfHeal: true

argocd confimap :

apiVersion: v1

data:

  admin.enabled: "true"

  application.instanceLabelKey: argocd.argoproj.io/instance

  exec.enabled: "false"

  helm.valuesFileSchemes: secrets+gpg-import, secrets+gpg-import-kubernetes, secrets+age-import,

    secrets+age-import-kubernetes, secrets,secrets+literal, https

  server.rbac.log.enforce.enable: "false"

  statusbadge.enabled: "false"

  timeout.hard.reconciliation: 0s

  timeout.reconciliation: 180s

  url: https://argocd.example.com

kind: ConfigMap

metadata:

  annotations:

    meta.helm.sh/release-name: argocd

    meta.helm.sh/release-namespace: argocd

  creationTimestamp: "2024-07-05T14:30:12Z"

  labels:

    app.kubernetes.io/component: server

    app.kubernetes.io/instance: argocd

    app.kubernetes.io/managed-by: Helm

    app.kubernetes.io/name: argocd-cm

    app.kubernetes.io/part-of: argocd

    app.kubernetes.io/version: v2.11.3

    helm.sh/chart: argo-cd-7.1.3

  name: argocd-cm

  namespace: argocd

  resourceVersion: "3889857"

  uid: ce560fb5-5800-4ca3-bde0-78293095eea7

error:

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = helm template . --name-template harish --namespace argocd --kube-version 1.29 --values <path to cached source>/helm-charts/platform/kube-prometheus-stack/values-dev.yaml --values secrets+age-import:///helm-secrets-private-keys/key.txt?values.sops.yaml <api versions removed> --include-crds failed exit status 1: No keys found in file [helm-secrets] Error while decrypting file: values.sops.yaml Error: plugin "scripts/run.sh downloader" exited with error

Sound like age does not find a valid decryption key in /helm-secrets-private-keys/key.txt but the error is not the same.

[priyas16]

Could it be possible that you miss an additional value property which is required in addition?

Since, we can see a difference between encrypted an non-encrypted. I expected that encryption works as expected.

Hi @jkroepke,

Only these two parameters (realm and accessToken) are required and they are working fine when we deploy the values.yaml manually. The issue occurs while using ArgoCD with the encrypted file reference.

Thanks!

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.