Support Vault for credential management

[engrun]

Hi Thanks for a great tool. We have just adopted this tool instead of managing everything by ourselves as docker images in AWS ECS. (as we do not want to invest in BOSH just for Concourse, so this project fits us perfectly!)

However, we would like to see Vault support for credential management, either as config parameters pointing to an existing vault installation, or even better, an optional flag to install Vault using bosh. There is already a boshrelease

We would be happy to assist in testing and providing feedback

[danyoung]

We had been thinking that if concourse-up is going to include credential management, this might be better served using credhub. The vision for concourse-up is for it to be a small, sharp tool. We also want to preserve the magic, non-interactive nature of the automation. How would you imagine the UX for vault initialisation and unsealing might work?

[engrun]

Hi. Unfortunately we have no prior experience with Vault, so how this actually might work I won't speculate about.

The reason for our interest in Vault is the Spring-Cloud stack, specifically the Spring Cloud Config server, which also supports Vault. So we thought backing both Concourse and Spring Cloud Config Server with Vault would make sense.

However, I appreciate the goal of tool (small, sharp, non-interactive).

Is there somehow a chance that you would consider providing the config parameters for Vault, and then we can install and manage it on our own. From the docs, it looks like we need to provide 5 parameters. In simplest form, could you just forward these if provided?

[engrun]

Hi. Is you possible to provide an update on your thoughts on credential management? Existing plans, roadmap etc.

If CredHub is a better fit for you, I guess this is viable to us (even though I think we would prefer Vault support first)

[danyoung]

Hi Rune,

Credhub support is fairly high in the backlog, but unfortunately we don't have anyone on the bench to implement this right now. I'll be sure to keep you updated once we start work on this. In the meantime, any information you can provide on how you'd like credential management to work would be useful

Thanks! Dan

[engrun]

Hi.

Not sure I understand what you mean by how credential management should work? Is this not given? Provide a server that concourse can use for creds. If you mean internally, under the hood, in concourse-up, I really should no advise on this.

However, I am reading the concourse docs on credential management, and would like concourse-up to support the flags/config parameters when running the concourse-up command, whether it is credhub or vault should be irrelevant. E.g.

concourse-up .....
--credhub-url https://10.2.0.3:9000 \
  --credhub-ca-cert /etc/my-ca.cert \
  --credhub-client-id =db02de05-fa39-4855-059b-67221c5c2f63 \
  --credhub-client-secret 6a174c20-f6de-a53c-74d2-6018fcceff64

BUT, I understand this solution is not your preferred one, which means concourse-up must provision the credhub/vault server itself as a part of concourse-up. Which for me breaks your vision for a "small, sharp tool" (do one thing, and do it well), as it then starts to grow into something bigger and more complex than just being a tool for concourse.

Hopefully this clarifies.

[engrun]

Do you have some kind of issue tracker, so that we can monitor the progress of this issue?

[danyoung]

You can see the status of the story for credhub support here: https://www.pivotaltracker.com/story/show/150128137

[engrun]

ok, thanks for the pointer. We have been discussing the need for credential management, so my guess is that we will probably roll out our own Vault instance pretty soon.

Are you willing to reconsider just supporting the Concourse config-flags instead of making vault/credhub a part of concourse-up?

[danyoung]

Yes I think we'd consider supporting config for external cred management. The next time we have people to work on this, we'll have a look.

[engrun]

Great! thanks

[danyoung]

@engrun We have a new story which isolates the Vault config support., which is in progress https://www.pivotaltracker.com/story/show/154018325

[engrun]

Super! thanks for info

[danyoung]

We have postponed the pass-through of vault configuration because only a publicly addressed Vault would be routable and we don't expect this to be useful for most users. Would be interested in your views on this. However, we have just delivered Credhub support, so Credhub is now included with the installation and collocated on the web instance in the latest release

[engrun]

Ok, thanks. Great work! We will check out Credhub as soon as we find the time.