Using `commit_bbl_state_dir` encourages committing secrets

[jasonkeene]

There are various credentials in bbl-state.json, .tfstate and deployment vars files. Having commit_bbl_state_dir in bbl-up and bbl-destroy encourages folks to use the git resource for these files. git was not designed to be a secret store. Being one push away from leaking director credentials is... risky.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/159270439

The labels on this github issue will be updated when the story is started.

[heyjcollins]

Hey Jason - thanks for bringing this up. There's a bunch of potential routes to take here and I could spin up a charter for the team to investigate but before doing so, I'd be curious to know your thoughts and ideas about how you'd go about solving this.

[rowanjacobs]

The cloudfoundry/bbl-state-resource may suggest an alternate path forward, using storage buckets on GCP.

[jasonkeene]

I'd like to see a flag to disable committing to git, or ideally one to enable it but that would break users.

Right now the error doesn't fail our jobs so it is just a bit of noise. @wfernandes wrote a vault-resource that tarballs up the bbl state dir and crams it into vault. That is how we are managing it.

I fear the git error message encourages using the git-resource to store bbl state dir, which, like I said previously can be risky.

Also, I have heard rumors of bbl supporting storing state securely using something like credhub and not hitting disk. I'm not fully up to speed with that development however.

[evanfarrar]

We don't currently have plans to store state in credhub. Bosh CLI may one day support secret storage during create-env, which would extend to BBL for free, but I don't believe that is under active development.

We do plan to eventually support storage in all major IaaS vendor's blob storage (and already do support it for S3) which would remove the temptation to use Git for this. You still have to opt in, but when opted in your state is always synced with the bucket and won't be on disk (outside of tmp) unless you pull it down.

Wherever you store your state, the critical items for ephemeral/integration/non-PII environments to not store on disk would be the IaaS credentials, which already can be stored in credhub (or Vault) via a Concourse's built in secret management. I think it is redundant to add the configuration and bosh credentials to Vault if you are a CF component team, these should be rotating on a daily basis.

[aegershman]

Haven't tried it yet, but what about something like git-crypt/sops? It's still committing to git, but encrypted with gpg keys. Not sure how well it'll work, I'm going to try it out in the next few days. Doesn't really solve the root concern since it's still in git, but figure I'd share the idea.

[davewalter]

Hi @jasonkeene

We recently made a change to the bbl-up and bbl-destroy tasks to allow users to optionally store the bbl-state as a tarball that can be persisted in S3/GCS/online storage instead of committed to a GH repo. If this is this a suitable solution, could you please close this issue so we know that we don't need to address it further in future?

Thanks, Dave

[aegershman]

^if using s3 for storing state as a tarball, is it possible to consume it as a tarball as input in other tasks?

[davewalter]

Hi @aegershman,

Yes. Both s3 and gcs resources support an unpack: true parameter that will unpack a tarball during the get step.

Regards, Dave

[Syerram]

@jasonkeene, Let us know if the last response from Dave solves the issue you described. I am closing due to inactivity but you can always reopen the issue if the problem still persists.