add credhub-colocated.yml for colocated CredHub

[brightzheng100]

This is a PR for adding colocated CredHub in Web VM integrated with Concourse as its credential management service.

To walk it through, please refer to below steps and make necessary changes to adapt to your env:

Preparation

Here we assume that:

1. We've enabled CredHub in BOSH Director for credential management;
2. We want to inject our TLS cert into Concourse (otherwise consider to enable cluster/operations/tls-vars.yml).

So we now prepare something like this:

$ ./connect-bosh-credhub.sh     # just to log into BOSH CredHub

$ credhub set -n /bosh-lite/concourse/atc_tls -t certificate \
  --private test.key.pem \
  --certificate test.crt.pem \
  --root ca.crt.pem

Spin Up Cluster

$ git clone https://github.com/brightzheng100/concourse-bosh-deployment.git
$ git checkout pr-credhub-colocated
$ cd concourse-bosh-deployment

$ bosh deploy -d concourse cluster/concourse.yml \
  -l versions.yml \
  -o cluster/operations/static-web.yml \
  -o cluster/operations/tls.yml \
  -o cluster/operations/privileged-https.yml \
  -o cluster/operations/add-local-users.yml \
  -o cluster/operations/uaa.yml \
  -o cluster/operations/storage-driver.yml \
  -o cluster/operations/credhub-colocated.yml \
  \
  -v web_ip=10.244.0.101 \
  -v external_url=https://concourse.test \
  -v network_name=default \
  -v web_vm_type=small \
  -v db_vm_type=small \
  -v db_persistent_disk_type=default \
  -v worker_vm_type=default \
  -v storage_driver=overlay \
  -v deployment_name=concourse \
  -v uaa_version='"62.0"' \
  -v uaa_sha1="4423bb02d1be2d3c2f212465f4598c9eb62a4b63" \
  -v credhub_version='2.1.2' \
  -v credhub_sha1='754a24dbffe8bc5efce7e698d935b5f4df541f38' \
  -v main_team_local_users='["admin"]' \
  -v add_local_users="[\"admin:Password1\"]"

SIT

Log Into The Colocated CredHub

The cluster/operations/credhub-colocated.yml has embedded UAA client creation for CredHub Admin named credhub_admin.

Use it directly for CredHub login:

$ credhub_admin_secret="$( credhub get -n /bosh-lite/concourse/credhub_admin_secret -j | jq -r .value )"

$ unset CREDHUB_SERVER && unset CREDHUB_CLIENT && unset CREDHUB_SECRET && unset CREDHUB_CA_CERT

$ credhub login -s https://10.244.0.101:8844 \
  --client-name credhub_admin \
  --client-secret="${credhub_admin_secret}" \
  --skip-tls-validation

$ credhub set -n /concourse/main/my-secret -t password -w MY-TOP-SECRET
id: cd7c0fc5-7eff-4f08-9539-f7d2900c5f71
name: /concourse/main/my-secret
type: password
value: <redacted>
version_created_at: "2019-03-08T08:52:06Z"

Test It Out

$ wget https://raw.githubusercontent.com/brightzheng100/concourse-demo/master/pipeline-parameterized-simple.yml
$ cat pipeline-parameterized-simple.yml
...
- name: hello-world
  plan:
  - task: say-hello
    params:
      MY_SECRET: ((my-secret))
...

$ fly login -t local -c https://concourse.test -k -u admin -p Password1
$ fly -t local sp -p pipeline-parameterized-simple -c pipeline-parameterized-simple.yml
$ fly -t local up -p pipeline-parameterized-simple
$ fly -t local trigger-job -j pipeline-parameterized-simple/hello-world -w
...
Successfully pulled busybox@sha256:061ca9704a714ee3e8b80523ec720c64f6209ad3f97c0ff7cb9ec7d19f15149f.

running sh -exc echo "MY_SECRET is ${MY_SECRET}"
+ echo 'MY_SECRET is MY-TOP-SECRET'
MY_SECRET is MY-TOP-SECRET
succeeded

Done! The ((my-secret)) in the pipeline is properly parsed and the actual credential is successfully retrieved during run time from its integrated colocated Credential Manager: CredHub.

[brightzheng100]

Already updated it for the compatibility of Concourse v5 and tested with the latest master. Kindly let me know whether it's useful for the community, @vito

[vito]

@brightzheng100 Sorry for the long wait. Happy to merge this in as it's fairly self-contained and it looks like you've been pretty thorough!