search

moxious / triage

testing triage actions for issues
0 stars 1 forks source link

Dashboards: Loading values takes a lot of time #403

Open tonypowa opened 1 week ago

tonypowa commented 1 week ago

What happened?

After I upgraded the grafana to version 11.1.x, the loading time was much longer than before(10.x).

I figured out that it's the values query that blocks the whole loading:

image

However, I don't see requests like this on the version under 11.x, also, I have tried both refresh On dashboard load or On time range change, the behavior keeps the same.

image

What did you expect to happen?

Can we keep the same behavior as 10.x before or refresh values simultaneously.

Because if you have a lot of values when values have a dependency. The blocking and cascading loading process will kill the user experience

Did this work before?

Yes, version under v11.x

How do we reproduce it?

  1. open a dashboard that have a lot of values

  2. view chrome debug portal

  3. the GET methods of values consumes the most part of time

Is the bug inside a dashboard panel?

No response

Environment (with versions)?

Grafana: 11.1.3

OS: windows

Browser: chrome

Grafana platform?

Kubernetes

Datasource(s)?

Prometheus

tonypowa commented 1 week ago

This issue is a test copy of an issue in another repo. Original issue: https://github.com/grafana/grafana/issues/#91612

moxious commented 1 week ago

Summary: User tonypowa reports a performance issue in Grafana version 11.1.x, where dashboard loading times are significantly longer compared to earlier versions, due to slow values queries. This behavior persists despite refresh settings and negatively impacts user experience, especially when dealing with many dependent values. The issue was not present in versions before 11.x.

moxious commented 1 week ago

Elaboration:

Hi @tonypowa, thank you for reporting this issue and providing details about the performance regression you've encountered after upgrading Grafana. To help us better understand and address the problem, could you provide us with a bit more information? Here are a few questions and suggestions that would be very helpful for our investigation:

  1. Could you provide the approximate number of panels and queries on the dashboard where you're experiencing the increase in load time?
  2. Do you see any errors or warnings in the Grafana server logs that could be related to the increased loading time?
  3. Is it possible for you to share the JSON model of the dashboard (with any sensitive information removed), so we can try to replicate the issue on our side?
  4. You mentioned that the 'values' query is where the bottleneck appears. Could you elaborate on what specific values or queries are being run? Providing the exact query or a similar non-sensitive example would be great.
  5. If it's feasible, could you create a minimal reproduction case with a smaller dashboard that also shows the slow loading times after the upgrade?
  6. Additionally, could you provide a screenshot of the Chrome Developer Tools Network tab showing the slow requests? It might give a clue as to what's happening during the data loading phase.
  7. Finally, have you had a chance to check the performance of the underlying data source itself? Sometimes issues can manifest within the visualization layer, but the root cause could be at the data retrieval level.

We understand that these requests might take some effort on your part, but they are essential to allow us to accurately diagnose the issue. Please let us know if you need any guidance on how to gather this information. Thank you for your contribution to the Grafana community!

moxious commented 1 week ago

Hi @tonypowa, thanks for bringing this to our attention. This seems to be related to performance issues with dashboards after an update. The right team to assist you would be the Backend platform project, as they handle core functionalities including performance optimization, which seems to have been affected by the upgrade.

Could you also provide specific examples of the query parameters and the time taken for these values to load compared to the previous version? This information will help the team in pinpointing the root cause of the slowdown.

Meanwhile, please check if any default settings related to dashboards were changed between versions that might impact the loading times.