feat: Supporting SOPS for helm.

[ichtestemalwieder]

Feature-Proposal: Please add support for SOPS and helm (by generic hooks/adabtibility of executed commands. This will solve lots of other problems/use cases too).

Motivation: Helm is becoming the defacto standard for packaging (besides kustomize). Especially sops is by far easiest and powerfull way to secure your secrets. Security is so important. It is currently not possible to use sops with helm in argoCD.

Problem: With a "Custom Mangement Plugin" I loose alle of the impotant GUI functionality and all of the great features of the already existing helm implementation.

Solutions: 1.) There seems to by no hooks to run after downloading a helm chart and before argocd does the templating (to allow decrypting with sops) => Would be an important feature 2.) There seems to be no way to alter the command that argoCD executes for the build. If this could be changed, on could integrate custom additional commands/pipe commands. => Would also be an important feature. 3.) Maybe the easiest and best solution is, hardcoding this as a new feature (with a simple GUI option) directly into argoCD. So argoCD fulfills the promise "Secure by default/Making security easy"

Thanks very much.

[jannfis]

Hi @ichtestemalwieder - have you seen https://argoproj.github.io/argo-cd/operator-manual/secret-management/ and the related discussions in https://github.com/argoproj/argo-cd/issues/1364?

I think @jessesuen is right that ArgoCD needs to stay unopinionated about secrets management, until maybe a defacto standard emerges.

Meanwhile, SOPS support for Helm could be integrated with a little more effort, but without using custom management plugin. I see there is https://github.com/futuresimple/helm-secrets, but it does not wrap around helm template yet (which ArgoCD uses to template out manifest files). So, probably one would have to get helm-secrets to also wrap secret decryption around the template command, and then install/configure helm-secrets plugin using either a custom build or an init-container.

[ichtestemalwieder]

@jannfis, thanks very much for your answer!

Yes I have read (before opening the issue) all the links. But even after spending several hours, I have not found a practical solution to get helm+sops working. And the problem with futuresimple/helm-secrets is, that that there were no commits in 9 month, besides lots of PRs, so in my eyes it's abandoned and furthermore does not work out of the box.

The reason for opening an issue: ArgoCD (this is not meant as a critique) has major limitations allowing one to simply embed custom security operations. And that would be needed to be addressed with high priority: Not only from the perspective, that security is so important, but also from a competitor/USP perspective. The current complexity (and limitations) around this, make argoCD quite useless respectively make other solutions (like jenkins-x with great preview environment, tekton and vault support) much more attractive.

So as stated initially, it would be so important to either have 1.) hooks or 2.) configurable CLI-Commands of the build in Render-Plugins, to easily inject custom security operations. Thanks very much.

[soupdiver]

I currently try to get the same running. However, I would disagree with so in my eyes it's abandoned and furthermore does not work out of the box. Using it daily and surely it works.

What I've done so far:

1. Create a custom image based on argoproj/argocd
2. Do RUN helm plugin install https://github.com/futuresimple/helm-secrets in Dockerfile
3. Update argocd-repo-server deployment to use my image

What's missing:

1. Setup the AWS environment inside the container to be able to access AWS KMS for en/decrypting
2. Use configManagementPlugins to overwrite generate with a wrapper that decrypts before calling helm template

Does this sound right @jannfis ?

[tiagomeireles]

I'm also running into issues getting this going. The latest version of helm-secrets supports wrapping helm template but requires tiller. I'm currently installing the helm-secrets and tillerless plugins.

I'm trying to figure out how to configure a plugin to use helm-secret like helm tiller run ci -- helm secrets template ${args...}.

[johnmarcou]

This should help to point to no tiller:

...
          env:
            - name: HELM_HOST
              value: notiller
...

[nemo83]

I would <3 to see this implemented.

If we follow the --values secrets.yaml approach we could have in the Application spec, something similar to:

    helm:
      parameters:
        - name: "my_param"
          value: "123"
      valueFiles:
        - values-whatever.yaml
      sopsSecrets:
        - secrets.yaml

or:

sops:
  secrets:
  - secrets.yaml

EDIT. Actually scratch that ^^ if we enable the secrets plugin... the helm templatewill do this w/o having to specify anything at application level. and hte secrets.yaml will just appear in the valueFiles list... apologies.

[yann-soubeyrand]

Here are the instructions on how we managed to integrate Sops with Helm and Argo CD without loosing the GUI functionalities: https://github.com/camptocamp/docker-argocd. We still have to build custom Docker images though, but it worked well as of now.

[lokiwins]

Hello! I was digging into this most of the day trying to get this working with https://github.com/futuresimple/helm-secrets which now wraps around helm template

I was able to get it to work with the following:

Created a custom build with sops and helm secrets plugin installed following the docs https://argoproj.github.io/argo-cd/operator-manual/custom_tools/#byoi-build-your-own-image

Created a config management plugin following https://argoproj.github.io/argo-cd/user-guide/config-management-plugins/

configManagementPlugins: |
  - name: helmSecrets
    generate:
      command: ["/bin/sh", "-c"]
      args: ["helm secrets template $VALUES_FILES . | sed '$d' | sed '$d'"]

I actually had to pipe the output to sed '$d' to get rid of some additional lines at the end of the output which are just paths for each of the decrypted secrets.yaml files. Otherwise it errors out with a YAML to JSON parsing error.

Then in the application spec specify the helmSecrets plugin

plugin:
  name: helmSecrets
  env:
    - name: VALUES_FILES
      value: "-f values.yaml -f ../../values/values.yaml -f ../../values/secrets/secrets.yaml -f ../../values/secrets/namespace/secrets.yaml"

I am sure there is a better way to get this working since helm-secrets supports wrapping helm template now. One thing you do lose compared to just using the helm plugin is the parameters section only shows the values files now instead of all the helm values.

Helm Version: 2.14.1 SOPS Version: 3.2.0 Helm Secrets installed via helm plugin install https://github.com/futuresimple/helm-secrets in Dockerfile

[kvaps]

git-crypt support would also be nice to have

[MPV]

An alternative could be to use something like kubernetes-external-secrets, which fetches your secrets into Kube from external secret backends, such as:

AWS Secrets Manager, AWS System Manager, Hashicorp Vault, Azure Key Vault, Google Secret Manager and Alibaba Cloud KMS Secret Manager.

[ittennull]

Hi guys, I created a project that lets Argocd use sops-encrypted secrets in Helm applications. https://gitlab.com/ittennull/sopshelm It's easy to install and works great!

[mbolek]

@lokiwins does your solution show proper values when trying to access "History and Rollback"? I get 404 and think it might be connected with the custom tool stuff. Could you please verify on your end? I guess it's because argocd doesn't really show the values anymore as you've mentioned

[lokiwins]

@lokiwins does your solution show proper values when trying to access "History and Rollback"? I get 404 and think it might be connected with the custom tool stuff. Could you please verify on your end? I guess it's because argocd doesn't really show the values anymore as you've mentioned

@mbolek I ended up moving away from requiring sops for our apps. Instead I decided to have a single chart to manage all the secrets. Secret changes are actually handled by a custom tool instead but for us changes to secrets don't happen too often so it all works out well. From there I modified all the helm charts to reference the kubernetes secret and just pull out the secrets I need for that particular app. Now I don't have to use secrets with argo and just use the default helm plugin.

[gogovan-vincentngai]

helm plugin install https://github.com/futuresimple/helm-secrets

Hi @lokiwins May i have your dockerfile ?i am stucking in here

[gogovan-vincentngai]

Here are the instructions on how we managed to integrate Sops with Helm and Argo CD without loosing the GUI functionalities: https://github.com/camptocamp/docker-argocd. We still have to build custom Docker images though, but it worked well as of now.

May i know is it same missing visualize on GUI?

I try swap the repo-server image to camptocamp/argocd:v1.8.5_c2c.1 and mount the secret

all deployment no longer showsup in GUI

[lokiwins]

helm plugin install https://github.com/futuresimple/helm-secrets

Hi @lokiwins May i have your dockerfile ?i am stucking in here

@gogovan-vincentngai Here is the Dockerfile I used back when I was testing this out. caution is very old. HELM_VERSION for this was 2.14.1

FROM argoproj/argocd:latest

# Switch to root for the ability to perform install
USER root

# Install tools needed for your repo-server to retrieve & decrypt secrets, render manifests 
# (e.g. curl, awscli, gpg, sops)
ARG HELM_VERSION
RUN apt-get update && \
    apt-get install -y \
        curl \
        awscli \
        gpg && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* && \
    curl -o /usr/local/bin/sops -L https://github.com/mozilla/sops/releases/download/3.2.0/sops-3.2.0.linux && \
    chmod +x /usr/local/bin/sops

RUN curl -o /helm-v${HELM_VERSION}-linux-amd64.tar.gz https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz && \
  mkdir -p /helm-v${HELM_VERSION}-linux-amd64 && \
  tar -zxvf /helm-v${HELM_VERSION}-linux-amd64.tar.gz -C /helm-v${HELM_VERSION}-linux-amd64

RUN mv /helm-v${HELM_VERSION}-linux-amd64/linux-amd64/helm /usr/local/helm
RUN mv /helm-v${HELM_VERSION}-linux-amd64/linux-amd64/tiller /usr/local/tiller

RUN rm -rf /helm-v${HELM_VERSION}-linux-amd64/
RUN rm -rf /helm-v${HELM_VERSION}-linux-amd64.tar.gz

# Switch back to non-root user
USER argocd
RUN helm plugin install https://github.com/futuresimple/helm-secrets
RUN helm secrets help

[andrei-dascalu]

If I'm not mistaking, sops is super-useful not just with Helm but also with Kustomize and plain manifests. It would be great to be supported out of the box.

Customizing the image and creating own wrappers is ok, but I'm working with several clusters and in some cases I have applications of all 3 types

[dshatokhin]

As anyone here trying to find best solution for storing secrets.

Maybe we can just use protocol approach to pass encrypted values files to standard helm plugin?

spec:
  source:
    helm:
      valueFiles:
      - values.yaml
      - secrets://secrets.yaml

or in UI:

[gogovan-vincentngai]

As anyone here trying to find best solution for storing secrets.

Maybe we can just use protocol approach to pass encrypted values files to standard helm plugin?

spec:
  source:
    helm:
      valueFiles:
      - values.yaml
      - secrets://secrets.yaml

or in UI:

Hi @dshatohin Luckly we have an external Vault Server as secret storage so my finially using https://github.com/IBM/argocd-vault-plugin plugin Works fine for me its easy and quick and more intuitive compare to sops But i need to download and made all helm chart so you can inject the vault path

I think if other stuff using sops will be the option .. but then you need to locallly encryption the password stuff and push to github however you can never known what is the password unless you have the key to dencryption

[congbang-le]

@lokiwins does your solution show proper values when trying to access "History and Rollback"? I get 404 and think it might be connected with the custom tool stuff. Could you please verify on your end? I guess it's because argocd doesn't really show the values anymore as you've mentioned

@mbolek I ended up moving away from requiring sops for our apps. Instead I decided to have a single chart to manage all the secrets. Secret changes are actually handled by a custom tool instead but for us changes to secrets don't happen too often so it all works out well. From there I modified all the helm charts to reference the kubernetes secret and just pull out the secrets I need for that particular app. Now I don't have to use secrets with argo and just use the default helm plugin.

After spending 3 days to integrate Helm secrets, sops and argocd, I ended up with your solution LOL :)

[jkroepke]

Maybe we can just use protocol approach to pass encrypted values files to standard helm plugin?

That would be the best way to do it.

After spending 3 days ..

I wrote a guide to integrate helm-secrets with ArgoCD.

https://github.com/jkroepke/helm-secrets/blob/main/docs/ARGOCD.md

Currently, it's not possible to fetch gpg keys other than local disk. This means, gpg keys still needs to be mounted at the argocd repo server.

[andrii-korotkov-verkada]

Hey. As of today, did sops become a de-factor standard for helm?