Stop generating global dashboard using state in specific grafana instance

[consideRatio]

This project's dashboards are jsonnet templates that render to grafana readable .json dashboard files.

One jsonnet dashboard template stands out though, the global dashboard. It needs to be generated using explicitly listed datasource names - names that are unique to the grafana instance. This is handled by the deploy.py script, which when asked to dpeloy the dashboards to a grafana instance first reads information about data sources, and then passes that to jsonnet when rendering the global dashboard.

I'd like us to transition away from generating the global dashboard using grafana instance specific state, because then:

1. We can pre-render all dashboards and publish them (#63)
2. Users of the global dashboard doesn't have to re-generate and re-deploy the dashboards if for example a datasource is added

The reason this was done in the first place is that it wasn't (and still isn't) possible to have a single panel providing info from a variable set of datasources.

About the impossibility of having a single panel querying X datasources

While its possible to repeat a panel or row of panels per datasource, it doesn't seem to be possible to repeat a query per datasource within a panel. - https://www.reddit.com/r/grafana/comments/pprmfp/repeating_query_per_multivalue_variable/?rdt=41004 - https://community.grafana.com/t/dashboard-datasource-include-all-not-working-to-query-multiple-dbs/91103/9?u=consideratio > **But all/multiselect of datasource doesn’t fit into current Grafana implementation.** Each query has reference e. g. A, which then can be used in expression, transformation. - https://community.grafana.com/t/repeat-queries-for-all-datasources/43490

I propose we accept this limitation, and pivot to provide multiple panels or rows of panels - one per datasource (one per prometheus-server / one per k8s cluster with hubs).

Proposal

We provide the ability to view one row of these four panels per datasource:

These four panels are already available in the "Jupyterhub dashboard" within a row, so in practice I suggest a modification there to support repeating these four panels and deleting the global dashboard entirely rather than providing the same stuff in two places.

What we currently have and the proposal replaces

[consideRatio]

@GeorgianaElena (initial developer) @choldgraf (known user) what do you think about replacing the current global dashboard with an ability to view multiple clusters hub usage like described above?

[choldgraf]

I don't understand the implementation side of this, so I will trust your + @GeorgianaElena's judgment when it comes to what is possible. I agree that any time we have to hard-code something, it becomes brittle, and I like the idea of finding ways to make things less brittle.

My main question is how I could answer the same question that (for me) the global dashboard answers:

• What is the trend-line of monthly, weekly, daily active users aggregated across all clusters/hubs/etc being deployed?
• Secondarily: Any other metric that might be important across all clusters (e.g. "What % downtime do we experience aggregated across all clusters?")

It sounds like perhaps the answer is: we shouldn't answer this with a single Grafana dashboard. If that's the case, perhaps we can provide guidance for how somebody could aggregate this data on their own, and then generate the aggregate graphs in a custom way? (e.g. that's similar to what we do in 2i2c at https://2i2c.org/kpis/cloud/