To keep current with the Prometheus Bosh releases and support newer versions of CF exports, the existing Prometheus Deployments need to be upgraded as follows:
[ ] Turn the two existing pipeline jobs for Production and Stage into four: Dev, Stage, Tooling, Prod
[ ] For the initial rollout - leave the existing deployments in tooling alone
[ ] Each deployment will be co-located on each BOSH Director
[ ] For the new tooling deployment - just colocate the needed BOSH and other exporters to support tooling alone - no CF needed
[ ] Remove the legacy cf-attic exporter and dashboards
[ ] Use the current supported cf-firehose export and dashboard
[ ] Migrate to using the existing upstream manifest
[ ] Add back in CG customizations to the deploy including making the new cf-firehose job be co-located on the prometheus VM instead of a dedicated VM
[ ] Very all the dashboards are getting the new data and working as expected
[ ] 14 days after prod and staging are deployed, remove the legacy deployments in tooling only leaving the new tooling specific Prometheus behind
To keep current with the Prometheus Bosh releases and support newer versions of CF exports, the existing Prometheus Deployments need to be upgraded as follows:
Notes
ADR on the approach with approval: https://github.com/cloud-gov/internal-docs/blob/main/docs/ADRs/ADR-0015-prometheus-location-change.md
Acceptance Criteria