Support kustomize helmCharts valuesFile

[Madic-]

I am using helmCharts with valuesFile to provide the configuration to the helm chart. But when I try to build the kubernetes manifests I get the following error:

Error: missing Resource metadata

kustomization.yaml

---
helmCharts:
  - name: checkmk
    repo: https://checkmk.github.io/checkmk_kube_agent
    version: 1.6.0
    releaseName: checkmk
    namespace: checkmk
    includeCRDs: true
    valuesFile: values.enc.yaml

generators:
  - kustomize-secret-generator.yaml

kustomize-secret-generator.yaml

---
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: checkmk-secret-generator
  namespace: checkmk
files:
  - values.enc.yaml

values.enc.yaml (decrypted)

clusterCollector:
    ingress:
        enabled: true
        className: traefik
        annotations:
            traefik.ingress.kubernetes.io/router.entrypoints: websecure
            traefik.ingress.kubernetes.io/router.tls: "true"
            traefik.ingress.kubernetes.io/router.middlewares: traefik-redirect@kubernetescrd
            kustomize.config.k8s.io/behavior: merge
            kustomize.config.k8s.io/needs-hash: "false"
        hosts:
            - host: checkmk-cluster-collector.k8s.example.com
              paths:
                - path: /
                  pathType: Prefix
        tls:
            - secretName: wildcard-cloudflare-production-01
              hosts:
                - checkmk-cluster-collector.k8s.example.com

I assume ksops decrypts the file before the values are send to kustomize? And it tries to find kubernetes manifest specific configuration parameters in the values.enc.yaml?

[dosubot[bot]]

Hey @Madic-! I'm here to assist you with any issues you encounter. I can help you solve bugs, answer questions, and become a contributor.

To resolve the "Error: missing Resource metadata" when using Helm charts with a valuesFile in your kustomization.yaml, ensure that your Helm chart templates include the necessary metadata fields (apiVersion, kind, and metadata). The error typically occurs when these fields are missing or incorrectly specified in the resources generated by the Helm chart.

Here is an example of how you might structure your kustomization.yaml to include a valuesFile:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

helmCharts:
  - name: checkmk
    repo: https://checkmk.github.io/checkmk_kube_agent
    version: 1.6.0
    releaseName: checkmk
    namespace: checkmk
    includeCRDs: true
    valuesFile: values.enc.yaml

generators:
  - kustomize-secret-generator.yaml

Ensure that the values.enc.yaml file and the Helm chart templates include the necessary metadata. For example, a Kubernetes resource template in the Helm chart should look like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-config
data:
  key: value

If the Helm chart templates are missing these fields, you will need to update them accordingly. If you are using a third-party Helm chart, you may need to fork the chart and add the necessary metadata fields.

Additionally, ensure that your kustomization.yaml is correctly configured to use the Helm chart and the valuesFile. If the issue persists, you may need to debug the Helm chart templates to identify which resource is missing the required metadata ^[1].

To continue talking to Dosu, mention @dosu.

[devstein]

Hey @Madic-

I assume ksops decrypts the file before the values are send to kustomize? And it tries to find kubernetes manifest specific configuration parameters in the values.enc.yaml?

Correct. Similar to what the bot suggested, all of the files passed to the files key in the generator manifest are expected to be valid K8s resources when decrypted.

files:
  - values.enc.yaml

I'm not sure what it would take to support helmChart.valuesFiles off the top of my head. I'd need to investigate further. If you have ideas, let me know! We welcome contributions

[Madic-]

Thank you for your answer. I can't provide code. Just ideas and test already written code.

Maybe extend the the ksops secret generator configuration with another option, e.g.helmvaluesfile, and treat it differently:

---
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: checkmk-secret-generator
  namespace: checkmk
files:
  - secret.enc.yaml
 helmvaluesfile:
  - values.enc.yaml

[Madic-]

I thought a bit about the issue and think I understand the problem a bit better. Kustomize itself executes helm to render the helm chart and then does it's own magic.

So one possibility could be, that ksops needs to get the manifests from helm, decrypts the secrets and then forwards the manifests to kustomize

[Madic-]

I did exactly that via shell script.

kustomization.yaml

---
helmCharts:
  - name: checkmk
    repo: https://checkmk.github.io/checkmk_kube_agent
    version: 1.6.0
    releaseName: checkmk
    namespace: checkmk
    includeCRDs: true
    valuesFile: values.enc.yaml.decrypted

generators:
  - kustomize-secret-generator-sops.yaml

kustomize-secret-generator-sops.yaml

---
kind: SopsDecrypt
metadata:
  name: sopsdecryptshell
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ./sops-decrypt.sh
files:
  - values.enc.yaml

sops-decrypt.sh

#!/bin/bash

# read the `kind: ResourceList` from stdin
RESOURCELIST=$(cat)

# Get the list of files to decrypt
FILES=$(echo "$RESOURCELIST" |
  awk '/functionConfig:/,0' |
  awk '/files:/,/metadata:/' |
  grep '\- ' |
  sed 's/- //' |
  tr -d ' ')

# Decrypt the files
for i in $FILES; do
  sops --decrypt --input-type yaml --output-type yaml "$i" >"$i.decrypted"
done

It's kind of working. I can't decrypt inplace the file because that would change it and would create problems with git. So it creates a file with the extension .decrypted. But even with that file, kustomize seems to read the values.enc.yaml.decrypted before the generator gets executed. Is there some way to run it before kustomize reads the values.enc.yaml?

[Madic-]

Because I wanted a solution, I investigated some more time. The previous way wasn't working for me. Because I'm using ArgoCD, I began reading docs from it and found ConfigManagementPlugins. I do know that at this point it's getting ouf of scope of ksops. But maybe it can help others or by designing a solution within ksops.

The Configmap which configures the ConfigManagementPlugin:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmp-sops-plugin
  namespace: argocd
data:
  plugin.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: cmp-sops-decrypt
    spec:
      version: v1.0
      generate:
        command: [sh, -c]
        args:
          - sops --decrypt --input-type yaml --output-type yaml values.enc.yaml > values.yaml;
            kustomize build --enable-helm --enable-alpha-plugins --enable-exec .
      discover:
        fileName: "values.enc.yaml"

If this plugin finds a values.enc.yaml file in the argo-cd app, argo-cd executes the cmp-sops-decrypt CMP which runs sops decrypting the file to values.yaml, and then runs kustomize.

The ConfigManagementPlugin needs to be run as a sidecar container for the argo-cd repo-server so the deployment of it needs to be extended. The helm values I adjusted:

repoServer:
  volumes:
    - name: custom-tools
      emptyDir: {}
    - name: sops-age
      secret:
        secretName: sops-age
    - name: cmp-tmp
      emptyDir: {}
    - name: cmp-sops-plugin
      configMap:
        name: argocd-cmp-sops-plugin
  initContainers:
    - name: install-ksops
      image: viaductoss/ksops:v4.3.1
      command:
        - /bin/sh
        - -c
      args:
        - echo "Installing KSOPS..."; mv ksops /custom-tools/; mv kustomize /custom-tools/; echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
    - name: install-sops
      image: ghcr.io/getsops/sops:v3.8.1-alpine
      command:
        - /bin/sh
        - -c
      args:
        - echo "Installing SOPS..."; cp /usr/local/bin/sops /custom-tools/; echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
    - name: install-helm
      image: alpine/helm:3.15.1
      command:
        - /bin/sh
        - -c
      args:
        - echo "Installing helm..."; cp /usr/bin/helm /custom-tools/; echo "Done.";
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools
  extraContainers:
    - name: cmp-sops-plugin
      command:
        - "/var/run/argocd/argocd-cmp-server"
      image: alpine:3.20.0
      imagePullPolicy: IfNotPresent
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /home/argocd/cmp-server/config/plugin.yaml
          subPath: plugin.yaml
          name: cmp-sops-plugin
        - mountPath: /tmp
          name: cmp-tmp
        - mountPath: /usr/local/bin/kustomize
          name: custom-tools
          subPath: kustomize
        - mountPath: /usr/local/bin/ksops
          name: custom-tools
          subPath: ksops
        - mountPath: /usr/local/bin/sops
          name: custom-tools
          subPath: sops
        - mountPath: /usr/local/bin/helm
          name: custom-tools
          subPath: helm
        - mountPath: /.config/sops/age
          name: sops-age
          readOnly: true
  volumeMounts:
    - mountPath: /usr/local/bin/kustomize
      name: custom-tools
      subPath: kustomize
    - mountPath: /usr/local/bin/ksops
      name: custom-tools
      subPath: ksops
    - mountPath: /.config/sops/age
      name: sops-age
      readOnly: true

This basically builds the plugin container with all required tools on-demand.

Of course, the configuration could be way shorter if a container, that already includes the following binaries, would be used 🤷

• kustomize (from ksops)
• ksops
• sops
• helm