Explore options for setting up the community owned stemcell concourse

[friegger]

Motivation

Explore options to setup a concourse that runs the community stemcell build.

Ideas

• using GCP should make member management easy
• which tool to use to setup concourse? How do the costs compare? What about ease of upgrade?
  • control-tower using bosh
  • on k8s
• how to manage credentials within the deployment? We want to avoid having to manage any credentials outside of the GCP account
  • CredHub
  • Vault
  • other secret management backend Concourse is supporting
  • no Concourse secret backend, instead use pipeline external credentials that are stored in GCP Secret Manager

Criteria

• the concourse setup should be cost effective and easy to maintain/update
• it should be easy to on- and off-board contributors
• any necessary credentials should be managed within the GCP account

ToDo

• [ ] K8s based PoC
• [ ] PoC based on https://github.com/EngineerBetter/control-tower

Links

• K8s PoC https://github.com/rkoster/bosh-community-stemcell-ci-infra (containing next step issues)
• control-tower PoC: https://github.com/StefanWutz/bosh-community-stemcell-ci-ct-infra

[yatzek]

bosh-community-stemcell-ci-infra has an option of worker pods autoscaling based on concourse metrics (https://concourse-ci.org/metrics.html). This is not yet implemented, we would have to use something like the prometheus adapter (https://github.com/kubernetes-sigs/prometheus-adapter) to expose metrics to the HPA (horizontal pod autoscaler). Not sure if this feature is really needed.

[yatzek]

Community concourse available at https://bosh.ci.cloudfoundry.org/ and deployed to GKE using https://github.com/cloudfoundry/bosh-community-stemcell-ci-infra. Closing this ticket.