[research: 3d] Figure out how to improve platform secrets management

[afeld]

In order to know how to reduce friction and risk from manual secrets-handling, we want to spend up to 3 days discussing and prototyping options for better secrets management.

Acceptance Criteria

• [ ] We can write stories for how we actually want to improve what we're doing and quantify the effort involved.

---

We, as a team, feel that there's a lot of room for how we manage secrets for the platform. The main areas are:

• BOSH secrets
• Concourse credentials

Our current practice includes:

• Credentials floating around on engineers' machines
• For Concourse, no persistent canonical source
• Secrets are tedious to view and update
• Secrets can't be updated atomically (i.e. it's easy to clobber other people's changes)

For both BOSH and Concourse, we need to figure out:

• What existing options are out there
• What other Concourse / Cloud Foundry operators are doing
• What other solutions are being worked on by other teams

@LinuxBozo Anything I missed?

@mogul Ideas about the best way to proceed? Do we do a kickoff meeting internally? Do we reach out to teams outside of 18F via various channels and see who's interested in having a little summit (or something) around this?

Relevant links

• https://github.com/starkandwayne/airmarshal/issues/1
• https://github.com/concourse/concourse/issues/291
• http://www.starkandwayne.com/blog/spruce-vault-concourse-you/
  • https://github.com/starkandwayne/genesis/issues/32
• https://github.com/cloudfoundry-community/vault-boshrelease

[rogeruiz]

Related to the work on #33

[afeld]

• [ ] Re-reframe this as a research sprint

[mogul]

From #general in the CF slack...

[datn]

I've begun looking into Vault and will try to generate some well groomed issues around steps that need taking about secrets management.

[jmcarp]

If we decide to mess with vault, I started https://github.com/jmcarp/cg-deploy-vault a while back.

[datn]

Thanks @jmcarp !

[datn]

Picking this back up, I am now running Vault locally to learn about it and see whether it's feasible to address the concern about unsealing in an automated environment that Josh brought up (which is touched on here and here.

[jmcarp]

WIP Proposal:

• Concourse workers can fetch vault secrets used in bosh deploys using ec2 auth.
  • Alternatively, use approle auth; means concourse jobs must be passed secrets explicitly, but allows for finer granularity between pipelines
• Concourse jobs must pass the correct client nonce in vault authentication
• Bosh deploy manifests fetch secrets from vault using the spruce vault operator
• Bootstrapping (based on https://starkandwayne.gitbooks.io/codex/content/aws.html#vault-init)
  • On creating a new environment, deploy a vault-init instance
  • Operator adds secrets to deploy master-bosh, tooling-bosh, main vault
  • After bootstrap, copy secrets from vault-init to main vault and destroy vault-init

Note: this proposal only touches bosh secrets. I'm thinking we might want to defer changes to concourse secrets until concourse adds native support for vault and other backends.

[mogul]

To be closed once an implementation story is documented.

[rogeruiz]

I'll add the implementation story today

[rogeruiz]

Closed by 18F/cg-product#657