Add an overlay to provision creds and configure mTLS for Prometheus

[mike1808]

To allow Prometheus to communicate with Istio sidecar injected pods it has to have required credentials. To provision these credentials we manually inject istio-proxy sidecar to the Prometheus server deployment generated by helm template cf-for-k8s-prometheus stable/prometheus -n cf-system --set server.podLabels.what\-am\-i=prometheus command. The proxy sidecar will generate the key and certificates and put them to /etc/istio-certs. Then we configure Prometheus config to use these certs for requesting metrics endpoints on the node.

To test this overlay you have to deploy Prometheus. Follow Prometheus installation guideline from cf-for-k8s-metric team but instead of using helm install use helm template and save the generated YAML to some file, then apply the overlay by ytt -f <prometheus.yaml> -f config/values.yaml -f config/provision-prometheus-certs.yaml

The overlay is based on:

• Provision a certificate and key for an application without sidecars
• Istio's Prometheus deployment template
• Istio's Prometheus configmap template

#174408928

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/174557817

The labels on this github issue will be updated when the story is started.