m-lab / prometheus-support

Prometheus configuration for M-Lab running on GKE
Apache License 2.0
19 stars 11 forks source link

Enable dashboards in source #133

Open stephen-soltesz opened 6 years ago

stephen-soltesz commented 6 years ago

If we cannot do automatic dashboard "sync" between staging & production, we could still setup read-only production instance of grafana and load dashboards from source. The process of exporting from staging and adding to source control could be manual.

The grafana configuration mechanism is the dashboards.json directive: http://docs.grafana.org/installation/configuration/#dashboards-json

stephen-soltesz commented 6 years ago

This looks like it will be possible.

The different projects should still allow different levels of access to encourage converting dashboards into code and getting them into staging & prod.

Proposal:

It appears to be possible to delete dashboards through the Grafana API. Dashboards created via the web UI have type dash-db. Every day, a job could scan for these and delete them from Staging, for example.

Also, we should probably flush sandbox dashboards periodically. E.g. after a dashboard has a creation date older than a month or the last modification time is older than a month. This would help sandbox stay manageable.

For example listing and showing a specific dashboard meta data:

curl http://admin:secret@localhost:3000/api/search | json_pp
[
   {
      "isStarred" : false,
      "id" : 0,
      "type" : "dash-json",
      "title" : "Frontend Stats2",
      "tags" : [],
      "uri" : "file/example.json"
   },
   {
      "tags" : [],
      "uri" : "db/native-dashboard",
      "isStarred" : false,
      "type" : "dash-db",
      "title" : "Native dashboard",
      "id" : 1
   }
]

curl http://admin:secret@localhost:3000/api/dashboards/db/native-dashboard | json_pp
{
   "meta" : {
      "createdBy" : "admin",
      "canStar" : true,
      "updated" : "2017-12-01T19:49:26Z",
      "created" : "2017-12-01T19:46:44Z",
      "canSave" : true,
      "updatedBy" : "admin",
      "expires" : "0001-01-01T00:00:00Z",
      "type" : "db",
      "slug" : "native-dashboard",
      "version" : 2,
      "canEdit" : true
   },
   ...
stephen-soltesz commented 6 years ago

This is an area of active development and consideration by the Grafana devs. https://github.com/grafana/grafana/issues/9957

So, be mindful of adopting conventions that may need to be redone or that could be future compatible.