Closed christopheblin closed 2 years ago
My current understanding is that the dashboard should be put inside a configmap and then loaded into grafana in order to survive restarts
Yes. This can be configured via: https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml#L492
Or, if you want to persist dashboards created via the frontend, enable persistence. See: https://github.com/grafana/helm-charts/blob/main/charts/grafana/values.yaml#L272
@monotek thanks for your answer
what I do not understand is how to put these config values inside the values.yaml of kube-prometheus helm chart (https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml)
as you can see, I can configure persistence for prometheus (though prometheus.prometheusSpec.storageSpec) but there is not this option in the grafana section (https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml#L620)
unless there is somehow a way to pass all the grafana/values.yaml directly in the grafana section ?
@monotek thanks for your answer
what I do not understand is how to put these config values inside the values.yaml of kube-prometheus helm chart (https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml)
Grafana is installed via a subchart. See: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/Chart.yaml#L46
So you can just use the normal values file, by adding stuff under grafana:
. See: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml#L622
@monotek thanks !
I just made a HUGE step to understand how this stack works thanks to your comment !
I'll try this asap and get back to you
I think my misunderstanding comes from https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#developing-prometheus-rules-and-grafana-dashboards
this section is only valid if you want to provide some dashboard/rule to the community and not for your own rules and dashboards (and in this case, this is why you have to go to the "upstream" repositories in order to have them synced in the next release)
now that makes a lot more sense to me !
I tried to configure persistence in grafana
At first, I tried with azurefile : it did not work (the error was service init failed: migration failed (id = create migration_log table): database is locked
and it matches multiple solutions like https://forums.rancher.com/t/persistent-storage-for-grafana-reason-service-init-failed-migration-failed-err-database-is-locked/19262
So, I tried with azuredisk : it works but everytime I "helm upgrade", 2 pods try to use the same disk and so it fails (rollbacking the changes)
Warning FailedAttachVolume 2m51s attachdetach-controller Multi-Attach error for volume "pvc-fc357abe-bfa8-4906-9079-db88b2430186" Volume is already used by pod(s) prometheus-operator-grafana-6c94c868b4-5qsl2
Warning FailedMount 48s kubelet Unable to attach or mount volumes: unmounted volumes=[storage], unattached volumes=[config sc-dashboard-provider storage prometheus-operator-grafana-token-ck8fz sc-datasources-volume sc-dashboard-volume]: timed out waiting for the condition
Finally, I tried to configure a postgres persistence : the chart does not support this out of the box (from my understanding), so I tried to mess up with the grafana.Ini section
grafana:
#configure the persistence
grafana.ini: |
[database]
type = postgres
host = xxx.privatelink.postgres.database.azure.com
name = xxx
user = grafana@xxx
password = xxx
but helm is not happy with that :(
coalesce.go:200: warning: cannot overwrite table with non table for grafana.ini (map[analytics:map[check_for_updates:true] grafana_net:map[url:https://grafana.net] log:map[mode:console] paths:map[data:/var/lib/grafana/ logs:/var/log/grafana plugins:/var/lib/grafana/plugins provisioning:/etc/grafana/provisioning]])
Error: UPGRADE FAILED: template: kube-prometheus-stack/charts/grafana/templates/deployment.yaml:36:28: executing "kube-prometheus-stack/charts/grafana/templates/deployment.yaml" at <include (print $.Template.BasePath "/configmap.yaml") .>: error calling include: template: kube-prometheus-stack/charts/grafana/templates/configmap.yaml:17:40: executing "kube-prometheus-stack/charts/grafana/templates/configmap.yaml" at <"grafana.ini">: range can't iterate over [database]
type = postgres
host = xxx.privatelink.postgres.database.azure.com
name = xxx
user = grafana@xxx
password = xxx
If possible, I'd like to know how other people are doing to configure persistence in the grafana chart on azure (first for custom dashboards and second for users) ?
Should I create another ticket or continue here ?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.
This issue is being automatically closed due to inactivity.
You need just create the configMap with
metadata:
labels:
grafana_dashboard: "1"
You have the option to store your dashboard configuration using a ConfigMap, incorporating the data from your <Exported_Dashboard>.json
. To ensure proper recognition by Grafana, label the ConfigMap with grafana_dashboard: "1"
. This approach is convenient for managing and deploying dashboard configurations.
It's worth noting that this functionality is activated by default in the kube-prometheus-stack.
Loading dashboards using a ConfigMap is cumbersome (with json->yaml conversion).. Also argocd thinks {{variable}} in json part is some function..
Is there way to make grafana to import dashboard from arbitrary URL (or stock grafana dashboards by ID) on pod start?
Is there way to make grafana to import dashboard from arbitrary URL (or stock grafana dashboards by ID) on pod start?
Yes, via the dashboards field, aka grafana.dashboards
in KPS.
BIG THANK YOU! I see it now!
Now I want to go even bolder! ))))
Is there way I can pull a dashboard directly from a different "primary" grafana (that has persistence)? I tried "Share->Link URL", but it requires browser-based login into that grafana.. Is there a way to specify credentials in the URL or token somehow in the grafana.dashboards?
The dashboards get downloaded by script download_dashboards.sh
defined in templates/_config.tpl
[1]. Configuration for the downloader is in the values file [2]. There are some examples of auth with token, bearer token, basic auth in the dashboards
field. You can also use Dashboard API.
[1] https://github.com/grafana/helm-charts/blob/0af99fac51424d0e5bb19e0da25a7750d3062f42/charts/grafana/templates/_config.tpl#L70 [2] https://github.com/grafana/helm-charts/blob/0af99fac51424d0e5bb19e0da25a7750d3062f42/charts/grafana/values.yaml#L189
Is your feature request related to a problem ?
I'm frustrated to lose my dashboard after the grafana pod has restarted
Describe the solution you'd like.
I'd like to create a dashboard in grafana and be able to use that dashboard after the pod restarts
So I'd like the doc to provide a small explanation "how to create your own additional dashboard"
Describe alternatives you've considered.
I read the doc https://github.com/prometheus-operator/kube-prometheus/blob/main/docs/developing-prometheus-rules-and-grafana-dashboards.md, but this is way too "high" for me
As a trial, I want to create a dashboard where I will have the gauges for all PV (instead of going to each PV through the dropdown in General / Kubernetes / Persistent Volumes)
At the moment, after
k rollout restart deployment prometheus-operator-grafana
, my dashboard is lostI did not find any way to configure a persistent storage for grafana
My current understanding is that the dashboard should be put inside a configmap and then loaded into grafana in order to survive restarts
Additional context.
No response