search

grafana / grafana

The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
https://grafana.com
GNU Affero General Public License v3.0
60.72k stars 11.61k forks source link

Grafana - could not resolve dashboards:uid:XxxXXX99x: Dashboard not found #87342

Open AllezWiggo opened 2 weeks ago

AllezWiggo commented 2 weeks ago

What happened?

Grafana fails to load a dashboard defined in a configmap. The sidecar is able to load the json to a /tmp folder: {"time": "2024-05-03T15:13:40.748693+00:00", "taskName": null, "msg": "Writing /tmp/dashboards/OpsDashboards/ops-dashboard.json (ascii)", "level": "INFO"} The json can be seen in the said location in the pod as well. But Grafana container throws following error: logger=provisioning.dashboard type=file name=sidecarProvider t=2024-05-03T16:49:28.773411742Z level=error msg="failed to save dashboard" file=/tmp/dashboards/OpsDashboards/ops-dashboard.json error="could not resolve dashboards:uid:XxxXXX99x: Dashboard not found"

This happens only for one dashboard json, other dashboards are loaded fine. The dashboard json hasn't changed recently and this used to work before.

What did you expect to happen?

Expected behaviour is the dashboard should get provisioned successfully.

Did this work before?

Yes, the dashboard used to get provisioned successfully before.

How do we reproduce it?

  1. Load the dashboard json in a configmap and use proper annotations. 
    apiVersion: v1
kind: ConfigMap
metadata:
name: "grafana-ops-dashboard"
labels:
grafana_dashboard: "1"
annotations:
grafana_dashboard_folder: "OpsDashboards"
data:
ops-dashboard.json: |
...
  2. Wait for sidecar to load the dashboard.
  3. Errors can be seen in grafana logs

Is the bug inside a dashboard panel?

No response

Environment (with versions)?

Grafana: v10.2.2 OS: Linux (Kubernetes v1.27.7) Browser: Chrome Version 124.0.6367.79 (Official Build) (x86_64)

Grafana platform?

Kubernetes

Datasource(s)?

Prometheus

babelfish commented 1 week ago

I'm experiencing the same problem, except in our case all of the dashboards are failing with this error message. Grafana started up and worked fine for around 13 hours then this error started happening simultaneously for all dashboards and all of the dashboards disappeared from the UI.

We're running Grafana 10.4.1 via v7.3.9 of the official helm chart.

AllezWiggo commented 1 week ago

Update: We were able to solve this issue by renaming the .json file in the configmap. Looks like the file could've been corrupted for some reason. Any way that seems to fix our issue, but it's worth investigating what is causing the file loading to fail and / or if any change is needed in design / impl. Thanks.