EPIC Self-hosting Grafana

[lucyb]

Questions to answer before we decide to self-host grafana

• [x] Is the official docker image safe to run in production?
  • [x] Does it run as root?
  • The official docker image uses Alpine and runs as the grafana user, not root.
• [x] How do we keep it up to date?
  • [x] can we create a Dockerfile (that basically just loads the upstream image), which we deploy to Dokku & which we can use dependabot to keep up to date?
• [x] How do we manage log ins? Manually.
  • [x] Can we setup SSO? Requires Grafana Enterprise plan (for self-hosted)
• [x] ~Is Dokku 4 the best place to host this?~ We should host on Dokku 3
  • Dokku 4 has access to the database
  • [x] what system resources does Grafana need? will it be a "noisy neighbour" for other services on Dokku 4?
  • [x] ~Note: Dokku 3 currently has access to the db cluster. Is this required or should we change it?~ separate ticket
  • As we're creating a read-only node, we can grant different access to that node, so we can remove access to the main node from dokku3 but keep access to the read-only node.
• [x] Can we query the production db safely?
  • Yes - by creating a read only node in the database cluster and by creating a read only user
• [x] Can we restrict access so that it doesn’t have access to more secure information, like the session table or oauth tokens? e.g can we give access to a limited set of tables? Can we create a view over the user table and give access to the view rather than the table itself? Yes (see ticket)
• [x] Will we need to update ansible?
• [x] Which repo do we use to store the dokku config for grafana?
  • opensafely-core/sysadmin/services
• [x] How to manage Grafana's state?
  • Grafana uses SQLite by default, but this involves storing state on the webserver and will be harder to manage in the long run. We should create a new database on the existing cluster instead. This will automatically setup backups for us.
• [x] Should we make the dashboards publicly available?
  • [x] Speak to Seb about this
  • [ ] Should we protect this behind cloudflare or something else?
  • [x] What** domain should we use? dashboards.opensafely.org??

Related issues:

• https://github.com/opensafely-core/job-server/issues/3644
• https://github.com/opensafely-core/sysadmin/issues/145
• https://github.com/opensafely-core/sysadmin/issues/146

[madwort]

self-hosted grafana is online at dashboards.opensafely.org, with CloudFlare proxy enabled on the hostname. Currently tidying up the process PR https://github.com/opensafely-core/sysadmin/pull/150

TODO: ensure the instance can send emails (invites, password reset etc)

[madwort]

some intermittent issues with accessing this services via dashboards.opensafely.org, although the dokku url works reliably.

[madwort]

ref. https://grafana.com/docs/grafana/latest/setup-grafana/configure-docker/ https://grafana.com/docs/grafana/latest/setup-grafana/installation/docker/

[madwort]

Disabling the CloudFlare proxy on the CloudFlare DNS setting page makes it work! 🎉

EDIT: after a bit of flip-flopping, which I think was due to DNS propogation

We may be able to use CloudFlare Cache Rules to enable caching & still have a functioning system.

[madwort]

need to merge PR, configure continuous deployment & ensure auto-merge of dependabot PRs.

[madwort]

nb. if you need to redeploy this before I'm back from holidays, you can just push the sysadmin repo at the grafana app on dokku3.

[lucyb]

@madwort I've noticed that we aren't getting dependabot alerts for the sysadmin repo at the moment. I know we have the dependabot workflow, but should we enable some of these settings too as part of this work, so that we are alerted when there are new versions of grafana or new vulnerabilities?

https://github.com/opensafely-core/sysadmin/settings/security_analysis

[madwort]

@lucyb yes, you're right, dependabot security updates and dependabot version updates are two essentially completely separate systems, we only have dependabot version updates configured. I have just enabled dependabot security updates for that repo - I think it doesn't have many config options.

[madwort]

todo:

• start reducing privilege levels for users
• add github module

[madwort]

I /think/ adding the github module is just running this line on dokku3 (& adding to the INSTALL.md)

dokku config:set grafana GF_INSTALL_PLUGINS="grafana-github-datasource"

I'll test later today.

[madwort]

TODO: continuous deployment, and auto-merging of the dependabot PRs

[madwort]

discussed with George yesterday, plan to keep the grafana prod deployment stuff in the metrics repo. Currently trying to figure out how to switch the grafana prod deployment from the current setup (the sysadmin repo, pushing git, building on dokku) to what I think is the best setup going forward (the metrics repo, building on CI, pushing a docker image). I'm going to set up a second grafana instance on dokku3 in order to do a dry-run of this change as I'm a bit unclear on how dokku will respond to these changes & don't want to break what is now effectively a prod system (with the wip dashboards!).

[madwort]

We have enabled public dashboards.

[madwort]

Continuous deployment is working, following https://github.com/ebmdatalab/metrics/pull/8

[madwort]

follow-up task https://github.com/ebmdatalab/metrics/issues/28

[madwort]

I think this is done now? Is there anything else? We could review user permissions?

[madwort]

all done