Application and Operational Metrics

[tmcgilchrist]

ocaml-docs-ci should report on application and operational level metrics via prometheus/grafana.

• [x] Production dashboard in grafana for https://docs.ci.ocaml.org
• [x] Staging dashboard in grafana for https://staging.docs.ci.ocaml.org
• [x] Setup ocluster connection to export stats (it already supports prometheus)
• [x] Add stat exports to docs-ci solver
• [x] Add stats export to monitor.ml.
• [x] Setup ocurrent to export stats (it already supports prometheus)
• [ ] Add host monitoring for storage and web server containers. Alert on disk space usage over 80%.
• [x] Add ocurrent support for exporting Node creation/update/destroy metrics (as a proxy for graph size)

Non-exhaustive list of tasks to cover the monitoring of ocaml-docs-ci application.

[tmcgilchrist]

@tmattio the only outstanding work is to add Alert on disk space usage over 80%.

[tmattio]

Operational metrics were added to the Grafana dashboard and we agreed that they are sufficient to monitor the services for the time being. We might add more in the future if we see that more metrics are necessary.