[grafana] persist home / dashboard settings

[JPFrancoia]

Hi,

I think I successfully installed Grafana on my k8s cluster. Here is the config specific to Grafana:

kube-prometheus-stack-values.yaml:

grafana:
  persistence:
    type: pvc
    enabled: true
    storageClassName: local-grafana
    accessModes:
    - ReadWriteOnce
    size: 500Mi
  extraVolumeMounts:
  - name: dashboards
    mountPath: /var/lib/grafana/dashboards
    hostPath: /home/djipey/grafana/dashboards
    readOnly: false

pv-grafana.yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-db
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  hostPath:
    path: /home/djipey/grafana
    type: DirectoryOrCreate
  storageClassName: local-grafana

I'm using local storage. It looks like it's working, when I ls the content of the grafana dir on the machine running the cluster:

❯ ls 
alerting  csv  dashboards  grafana.db  plugins  png

I can then access Grafana in my browser, and I can see the home page, the default dashboards, etc. However, if I change the home page (by removing the default panels for example) or if I star or change a dashboard, I lose the settings when I restart the grafana deployment, i.e: no dasboard will be starred and the default stock panels will be back on the home page.

Is there a way to avoid that?

[ripkort]

I've found same behavior with starred dashboards.

If the dashboard provisioned from the ConfigMap and I starred it, after POD restart it will not be present in starred dashboards.

If the dashboard was manually added and I starred it, it will remain the same after POD restart.

I think the issue is - after each Grafana POD restart, dashboards from ConfigMaps will be reloaded and will get new id in SQLite database. From the SQLite DB I found the following migration from migration_log table 91|delete stars for deleted dashboards|DELETE FROM star WHERE dashboard_id NOT IN (SELECT id FROM dashboard)|1||2022-08-31 17:53:01

Here is starred dashboards id 22654 before Grafana POD restart sqlite> SELECT * from star; 16|3|22654

And below same dashboard (after starred again) but with different id 22881 after POD restart sqlite> SELECT * from star; 17|3|22881

So having in mind that dashboards changed their ids after reload, the query above will delete starred dashboards.

So far I don't know how to bypass this issue.

P.S. tried latest helm chart which is 7.1.0 and issue is present there.