Sops and helm usage proposal

[jjangga0214]

Hi, thank you very much for writing a great book!

I'd like to make a proposal (or possibly question or discussion) on sops and helm usage for feeding an applications' secret file to k8s secret object.

Currently hello-sops example places staging-secrets.yaml and production-secrets.yaml inside of the demo chart.

k8s
 └── demo
     ├── <others are omitted for brevity>
     ├── Chart.yaml
     ├── production-secrets.yaml
     └── staging-secrets.yaml

However, the book suggests (in an applicable situation) making one single secret file (not k8s secrets object, but something like .env or production-secrets.yaml) as a strategy of avoiding secret duplication across applications.

Let's say I have multiple charts like demo1 and demo2. Also assume I have a secret file production-secrets.yaml, which has secrets for both demo1 and demo2. Unlike hello-sops, I want to keep it independent from each chart, so place it outside of the charts.

k8s
 ├── demo1
 │     ├── <omitted for brevity>
 │     └── Chart.yaml
 ├── demo2
 │     ├── <omitted for brevity>
 │     └── Chart.yaml
 └── production-secrets.yaml

However, .Files.Get cannot read an external file outside of the chart (There's a discussion though, helm#3276).

Thus, after decrypting production-secrets.yaml, I have to copy and paste the decrypted file under demo1 and demo2.

This is cumbersome.

Fortunately, using --set-file and .Values instead of .Files.Get can solve the problem.

For instance,

helm upgrade --install \
  --set-file mysecrets=./k8s/decrypted-production-secrets.yaml \
  production-demo ./k8s/demo 

# secrets.yaml in helm chart
data:
  mysecrets.yaml: {{ .Values.mysecrets | b64enc }}

helmfile supports --set-file as well. The below is same as the above.

releases:
- name: production-demo
  chart: k8s/demo 
  set: # --set-file mysecrets=./k8s/decrypted-production-secrets.yaml
  - name: mysecrets
    file: k8s/decrypted-production-secrets.yaml

I feel this way is more practical, as secrets are now "free" from charts.

I made a simple demonstration repo jjangga0214/k8s-sops-helm, so you might take a look.

How do you think of this?

Thanks.

[domingusj]

Thank you for opening the issue and starting the discussion! I think your idea is a great one and thanks for making the repo with the working example. I had a few questions to keep the discussion going and wondering if you have thoughts:

Are you typically managing all of your charts for multiple apps in the same repo? The secrets/file repo structure issue you point out is still going to be something to solve, but as a side note I've been happy lately with https://github.com/fluxcd/flux in cases where you want a single control repo for managing multiple apps.

I think there could be some upcoming changes to Flux based on their recent announcement to collaborate with Argo on the "GitOps" front (https://www.weave.works/blog/argo-flux-join-forces) so it's something I'll be watching and sort of looking to see what sorts of examples they come up with for secrets management and repo structure across apps/environments/clusters.

I have mostly worked in situations where each app has its own separate repo and the k8s/helm config for that app lives inside that same repo as the app, so in your example the chart for demo1 would live in a k8s folder in the demo1 app repo, and the chart for demo2 would live in the demo2 repo. In cases where secrets and/or config values are shared between them, then you either have to decide if you would rather duplicate them in 2 repos (and remember that if you change it in 1 you need to also change it in the other, which is less than ideal), or add some complexity to your deployment pipeline to bring in the secrets from some other place. Your example of a top-level shared folder in git is one of the simpler/better suggestions I have seen.

A 2nd strategy I have seen and used with some success is storing all secrets for all apps encrypted in an external place, such as a centralized encrypted git repo just for secrets, file(s) in a GCS/S3 bucket, or something similar. That path requires adding steps to pull in the appropriate secrets at deploy time and definitely adds some complexity. It requires coming up with a plan for updating/changing secrets, as well as figuring out a strategy for versioning across environments and apps.

At some point if a team decides to go down that path it's probably worth spending a little bit of time evaluating and deciding if it would just be worth taking the plunge with something designed for centralized secrets management, such as Hashicorp Vault. AWS, GCP, and Azure all have their own options for similar hosted secrets managers. Again, the sops path and your approach is much simpler than something like that, but at some point I think it makes sense for teams to evaluate if their situations is complex enough where it makes sense to add something like Vault to their environment.

Another similar situation to this that I've worked in is with subcharts/child charts. In the cases where apps need to share a value between them, Helm has the concept of Global values: https://helm.sh/docs/topics/chart_template_guide/subcharts_and_globals/. Your example doesn't quite seem like demo1 and demo2 are dependencies of each other so this may not exactly be relevant to your example. The tricky part I've found with globals in Helm charts is if a value needs to move from non-global to global then you have to remember to update any places in the chart that refer to it.

I have an example of that setup in a demo repo here: https://github.com/domingusj/rails-k8s-demo/blob/master/k8s/rails-k8s-demo/templates/secrets.yaml that handles cases where some shared secrets are needed by subcharts.

Thanks again for opening the issue and would love to keep the discussion going! Have a great New Years!

[domingusj]

Closing to clean up old issues, but reply back if you think this is unresolved and want to continue discussing.