search

getsops / sops

Simple and flexible tool for managing secrets
https://getsops.io/
Mozilla Public License 2.0
16.58k stars 865 forks source link

Support HCL files #292

Open sixfeetover opened 6 years ago

sixfeetover commented 6 years ago

Would love to see support for HCL. HCL is used for a lot of configuration files as part of the Hashicorp stack (Vault, Terraform, etc).

jvehent commented 6 years ago

We don't use HCL at Mozilla, so it's unlikely we'll write support for it. However, we're happy to help review patches from the community.

negz commented 6 years ago

Note that all HCL can be represented as JSON (i.e. you can configure at least Terraform using JSON rather than HCL). We use sops to decrypt JSON files and pass those to Terraform, i.e.:

terraform apply -var-file=<(sops -d secret.tfvars.json)

There are also some HCL to JSON (and vice versa) converters around that might help workaround this use case.

carlpett commented 6 years ago

@sixfeetover @negz As another Terraform solution, I just created https://github.com/carlpett/terraform-sops, which lets you use a sops-encrypted file as a data source

sixfeetover commented 6 years ago

@carlpett Oh, very clever 👍

jvehent commented 6 years ago

This is awesome! @autrilla: could we host this under sops/terraform and add a use-case examples in the examples folder?

autrilla commented 6 years ago

@jvehent yeah, that'd be great, as long as @carlpett is OK with having their work go in our repo.

starkers commented 6 years ago

I'd really like to use sops for .tfvars files specifically..

Although yes its perfectly possible to implement a fancy hcl2json > sops process its just ugly and because of json we loose the ability to put comments in etc..

carlpett commented 6 years ago

@autrilla I'd be perfectly fine with that, but I think a better goal would be to have it hosted in the "official third-party" organization terraform-providers. That way, it'll be possible to download automatically, rather than doing a manual install.

@starkers I actually started off by looking at that, but found this approach seemed easier to use to me. But maybe I missed something! How was your intended workflow?

negz commented 6 years ago

@starkers Take a look at https://github.com/mozilla/sops/issues/292#issuecomment-360985371 - Terraform will happily load JSON encoded tfvars files.

From https://github.com/hashicorp/hcl:

The API for HCL allows JSON as an input so that it is also machine-friendly (machines can generate JSON instead of trying to generate HCL).

autrilla commented 6 years ago

@carlpett yeah, that sounds better. Do you know what the process looks like to get it included there?

so0k commented 6 years ago

@carlpett thanks for sharing, as we were already using mozilla/sops - your provider allows our infra team to easily git clone and terraform apply without having to worry about custom make rules or tfvars

I did have to do the following to get this working on OSX:

go get github.com/carlpett/terraform-sops
mv $GOPATH/bin/terraform-sops $HOME/.terraform.d/plugins/terraform-provider-sops
abeluck commented 5 years ago

@carlpett any progress on getting terraform-sops hosted with terraform-providers? What needs to be done to make that happen?

carlpett commented 5 years ago

Thanks for poking on this @abeluck, I had let it slip for quite a while. Back when this discussion was started, the "Terraform Provider Development Program" was just about to start, and there was no clear way to enroll in it. I see now though that it is possible to sign up via a web form, so I'll get on it :) If Hashicorp agrees it is useful, there will be a bit of reviewing and adapting things (mainly repo structure, I think) so they mesh well with their automation.

binlab commented 4 years ago

It will be nice to have

robertwe commented 2 years ago

hi everyone, do we have any update on this issue?

abeluck commented 2 years ago

@robertwe

sops has no builtin support for HCL files.

If you're using terraform you can use @carlpett's terraform-provider-sops to load sops files (json or yaml) into terraform. It works very well.

Also shout out to terragrunt which lets you decrypt sops files as part of your terraform+terragrunt workflow.

robertwe commented 2 years ago

@abeluck thx for info. I'm aware of that. I also considered this option. There is also a possibility to load yaml file directly in terraform. The deal is I was searching for some super simple solution.

iT3E commented 1 year ago

terraform-provider-sops is great, but what about the rest of the Hashicorp stack? I'm struggling at finding a way to integrate sops with Packer..

phisch1991 commented 2 months ago

Note that all HCL can be represented as JSON (i.e. you can configure at least Terraform using JSON rather than HCL). We use sops to decrypt JSON files and pass those to Terraform, i.e.:

terraform apply -var-file=<(sops -d secret.tfvars.json)

There are also some HCL to JSON (and vice versa) converters around that might help workaround this use case.

Hi @negz, I know your answer is 6y old, however I wondered if you can help with an issue I am facing. I had the same idea to use process substitution. We ran into a problem with the way how terraform interpretes the input data. If the filename does not have a .json suffix, it will be interpreted as a hcl file, so it was not possible to pass a file which gets interpreted as json via process substitution. Did you manage to solve this issue? I could not find a way to tell Terraform how to interprete the file besides the filename.