automation to measure a baseline for soak tests

[jamieklassen]

As part of #2874 we should have a consistent way of creating a fresh 4.2.2 environment and seeding, soaking and measuring baseline metrics levels. I think this should amount to something like

1. Creating and/or zeroing out an influxdb database and grafana dashboard (this should be idempotent probably)
   • (assume that another sub-task of #2874 involves the creation of a bosh-deployed Concourse instance feeding metrics to the above influxdb and yet another task involves seeding that instance with data)
2. On demand (once the instance is demed 'soaked'), measuring 'one standard deviation above the mean' (or whatever other threshold we decide for 'statistical significance') and creating a threshold in Grafana of that value -- optionally with a Slack alert?

notes

• Assume for the time being that there will be only one influxdb database and grafana (being fed by a single Concourse instance) with hardcoded names. Over time, we may have multiples, probably under the control of a pool resource or similar abstraction.
• Unknowns include specifics of InfluxQL for writing the measurement query, and knowledge of Grafana APIs to programmatically create/modify dashboards and thresholds.

[jamieklassen]

helpful functions: MEAN, STDDEV, e.g.

> select mean(value) + stddev(value) from "destroying volumes to be garbage collected"
name: destroying volumes to be garbage collected
time mean_stddev
---- -----------
0    2.0462366010283515

[ddadlani]

Relevant context from @pivotal-jamie-klassen:

Branch with spike to add metrics: https://github.com/concourse/concourse/tree/hack/2874

Wiki page: https://github.com/concourse/concourse/wiki/Adding-a-new-instance-to-the-metrics-site