Grafana plugins should be customizable

[gdemonet]

Component: build, monitoring

Current situation:

In recent versions (6+), grafana/grafana container image doesn't come with the grafana-piechart-panel plugin installed. To make sure we had this plugin installed by default in our deployment without requiring an online installation at runtime, we baked our own image with this plugin installed at build-time.

This approach is sufficient for handling the dashboards we embed with MetalK8s releases. It will not be sufficient for handling user-provided dashboards or anything shipped with Solution archives (see #1860).

Proposed solution:

To be detailed, but first investigations led to the following conclusions:

• using ConfigMaps for such plugins isn't possible, due to the 1MB limit imposed by etcd
• serving the plugins through the repositories Pod directly from product ISOs could do the trick, as Grafana allows retrieving as ZIP archives from arbitrary URLs

[NicolasT]

I'm not sure we should go as far, at least for now. The plugins required for dashboard included in MetalK8s can be included as they are now. For Scality solutions, they can be added per request in a MetalK8s release (do we foresee a lot of them would be needed?).

[gdemonet]

I'm more concerned for users willing to deploy/build their own dashboards (and have no clue how many plugins would be required).

[thomasdanan]

We need to have a proper solution to manage user custos (like creating dashboards) without requesting additional CLI steps. It seems that the best approach to have customization persistence, HA, and maybe leverage the Grafana annotation feature is having a real DB behind Grafana (to be confirmed)

[gdemonet]

@thomasdanan This issue is about custom plugins, which is a different topic from what you are discussing here, which is persisting changes done in Grafana UI. Let's open a different ticket for tracking this (edit: opened #2851).