[Chore]: Deploy concourse

[georgethebeatle]

Background

We've been wanting better CI visibility for a long time now, so that we can have a better picture of CI stability and so that we can use tools like the flake hunter to chase flakes. These days we are facing even more problems because of the general slowness of github actions, which has made our PR turnaround painfully slow. Added to all of these concerns is the lack of confidence due to not running our e2e tests against real world clusters.

Action to take

All the pain points and concerns described above can be mitigated by deploying a concourse instance to run our tests. So let's take the following actions:

1. Create a k8s cluster to run our concourse instance
   • reuse whatever makes sense from eirini's scripts: https://github.com/cloudfoundry/eirini-private-config/blob/master/concourse-deployment/create-cluster.sh.
2. Create a concourse instance
   • reuse eirini's scripts: https://github.com/cloudfoundry/eirini-private-config/blob/master/concourse-deployment/upgrade-concourse.sh
   • All resources should be allocated in the CF-on-K8s-WG gcp project. All team members should already have invitations to that project.

     Impact

There is a single place where we can observe project health and run complex automations.

[kieron-dev]

PR in cf-k8s-secrets repo: https://github.com/cloudfoundry/cf-k8s-secrets/pull/1