[kube-prometheus-stack] Document how to add a custom dashboard

[christopheblin]

Is your feature request related to a problem ?

I'm frustrated to lose my dashboard after the grafana pod has restarted

Describe the solution you'd like.

I'd like to create a dashboard in grafana and be able to use that dashboard after the pod restarts

So I'd like the doc to provide a small explanation "how to create your own additional dashboard"

Describe alternatives you've considered.

I read the doc https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/developing-prometheus-rules-and-grafana-dashboards.md, but this is way too "high" for me

As a trial, I want to create a dashboard where I will have the gauges for all PV (instead of going to each PV through the dropdown in General / Kubernetes / Persistent Volumes)

At the moment, after k rollout restart deployment prometheus-operator-grafana, my dashboard is lost

I did not find any way to configure a persistent storage for grafana

My current understanding is that the dashboard should be put inside a configmap and then loaded into grafana in order to survive restarts

Additional context.

No response

[monotek]

My current understanding is that the dashboard should be put inside a configmap and then loaded into grafana in order to survive restarts

Yes. This can be configured via: https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml#L492

Or, if you want to persist dashboards created via the frontend, enable persistence. See: https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml#L272

[christopheblin]

@monotek thanks for your answer

what I do not understand is how to put these config values inside the values.yaml of kube-prometheus helm chart (https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml)

as you can see, I can configure persistence for prometheus (though prometheus.prometheusSpec.storageSpec) but there is not this option in the grafana section (https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml#L620)

unless there is somehow a way to pass all the grafana/values.yaml directly in the grafana section ?

[monotek]

@monotek thanks for your answer

what I do not understand is how to put these config values inside the values.yaml of kube-prometheus helm chart (https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml)

Grafana is installed via a subchart. See: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/Chart.yaml#L46

So you can just use the normal values file, by adding stuff under grafana:. See: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml#L622

[christopheblin]

@monotek thanks !

I just made a HUGE step to understand how this stack works thanks to your comment !

I'll try this asap and get back to you

I think my misunderstanding comes from https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#developing-prometheus-rules-and-grafana-dashboards

this section is only valid if you want to provide some dashboard/rule to the community and not for your own rules and dashboards (and in this case, this is why you have to go to the "upstream" repositories in order to have them synced in the next release)

now that makes a lot more sense to me !

[christopheblin]

I tried to configure persistence in grafana

At first, I tried with azurefile : it did not work (the error was service init failed: migration failed (id = create migration_log table): database is locked and it matches multiple solutions like https://forums.rancher.com/t/persistent-storage-for-grafana-reason-service-init-failed-migration-failed-err-database-is-locked/19262

So, I tried with azuredisk : it works but everytime I "helm upgrade", 2 pods try to use the same disk and so it fails (rollbacking the changes)

Warning  FailedAttachVolume  2m51s  attachdetach-controller  Multi-Attach error for volume "pvc-fc357abe-bfa8-4906-9079-db88b2430186" Volume is already used by pod(s) prometheus-operator-grafana-6c94c868b4-5qsl2
Warning  FailedMount         48s    kubelet                  Unable to attach or mount volumes: unmounted volumes=[storage], unattached volumes=[config sc-dashboard-provider storage prometheus-operator-grafana-token-ck8fz sc-datasources-volume sc-dashboard-volume]: timed out waiting for the condition

Finally, I tried to configure a postgres persistence : the chart does not support this out of the box (from my understanding), so I tried to mess up with the grafana.Ini section

grafana:
  #configure the persistence
  grafana.ini: |
    [database]
    type = postgres
    host = xxx.privatelink.postgres.database.azure.com
    name = xxx
    user = grafana@xxx
    password = xxx

but helm is not happy with that :(

coalesce.go:200: warning: cannot overwrite table with non table for grafana.ini (map[analytics:map[check_for_updates:true] grafana_net:map[url:https://grafana.net] log:map[mode:console] paths:map[data:/var/lib/grafana/ logs:/var/log/grafana plugins:/var/lib/grafana/plugins provisioning:/etc/grafana/provisioning]])
Error: UPGRADE FAILED: template: kube-prometheus-stack/charts/grafana/templates/deployment.yaml:36:28: executing "kube-prometheus-stack/charts/grafana/templates/deployment.yaml" at <include (print $.Template.BasePath "/configmap.yaml") .>: error calling include: template: kube-prometheus-stack/charts/grafana/templates/configmap.yaml:17:40: executing "kube-prometheus-stack/charts/grafana/templates/configmap.yaml" at <"grafana.ini">: range can't iterate over [database]
type = postgres
host = xxx.privatelink.postgres.database.azure.com
name = xxx
user = grafana@xxx
password = xxx

If possible, I'd like to know how other people are doing to configure persistence in the grafana chart on azure (first for custom dashboards and second for users) ?

Should I create another ticket or continue here ?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale[bot]]

This issue is being automatically closed due to inactivity.

[glavk]

You need just create the configMap with

metadata:
  labels:
    grafana_dashboard: "1"

Example - https://docs.syseleven.de/metakube-accelerator/building-blocks/observability-monitoring/kube-prometheus-stack#adding-grafana-dashboards

[ashkankamyab]

You have the option to store your dashboard configuration using a ConfigMap, incorporating the data from your <Exported_Dashboard>.json. To ensure proper recognition by Grafana, label the ConfigMap with grafana_dashboard: "1". This approach is convenient for managing and deploying dashboard configurations.

It's worth noting that this functionality is activated by default in the kube-prometheus-stack.

[skliarie]

Loading dashboards using a ConfigMap is cumbersome (with json->yaml conversion).. Also argocd thinks {{variable}} in json part is some function..

Is there way to make grafana to import dashboard from arbitrary URL (or stock grafana dashboards by ID) on pod start?

[zeritti]

Is there way to make grafana to import dashboard from arbitrary URL (or stock grafana dashboards by ID) on pod start?

Yes, via the dashboards field, aka grafana.dashboards in KPS.

[skliarie]

BIG THANK YOU! I see it now!

Now I want to go even bolder! ))))

Is there way I can pull a dashboard directly from a different "primary" grafana (that has persistence)? I tried "Share->Link URL", but it requires browser-based login into that grafana.. Is there a way to specify credentials in the URL or token somehow in the grafana.dashboards?

[zeritti]

The dashboards get downloaded by script download_dashboards.sh defined in templates/_config.tpl[1]. Configuration for the downloader is in the values file [2]. There are some examples of auth with token, bearer token, basic auth in the dashboards field. You can also use Dashboard API.

[1] https://github.com/grafana/helm-charts/blob/0af99fac51424d0e5bb19e0da25a7750d3062f42/charts/grafana/templates/_config.tpl#L70 [2] https://github.com/grafana/helm-charts/blob/0af99fac51424d0e5bb19e0da25a7750d3062f42/charts/grafana/values.yaml#L189