search

neuralfraud / grafana-prtg

A PRTG Datasource plugin for Grafana
180 stars 75 forks source link

Very slow when querying large amounts of data #138

Open forever765 opened 5 years ago

forever765 commented 5 years ago

Grafana Version: 6.4.2 Grafana-PRTG Plugin Version: 4.0

Hi there,

The grafana-prtg plugin is very slow when querying large amounts of xml data. For example, I have a dashboard that shows the NIC traffic graph. This dashboard contains 2 grafana graph panels. Each panel contains 2 channels (inbound and outbound traffic).

The time range of the dashboard is "last 6 hours". When I refresh the dashboard, the first panel takes about 10 seconds to display, and the two panels fully display about 20s. This is really unbearable.

I use the Query Inspector of the grafana panel to check. In most cases, I can see the results of the query, but after a few seconds, chrome will prompt for a crash. The recurrence rate of chrome crash is 100%.

Using the same request as the grafana-prtg plugin, I found that the single channel xml file returned by prtg is 750KB (about 11,000 lines), because each panel has two channels, so a panel needs to process about 1.5MB of xml data. PRTG is set to collect data once in 30 seconds, so the amount of data in 6 hours is very large.

I want to know what the root cause of the problem is, and what can be done to increase the loading speed? In addition, is the grafana-prtg plugin sending and processing requests concurrent?

At last, I am very glad to see that the author is still maintaining this project, and I sincerely thank you for your contribution.

neuralfraud commented 5 years ago

I was too lazy to try and parse XML nodes so I used XSLT to convert it into JSON. That is somewhat slow, but I've never seen a dashboard load that slowly. What is your query? click the query inspector button on one of those queries and paste in a sample of the data.. something seems awfully strange with the amount of data you say you're getting back considering that's only 720 datapoints.

forever765 commented 5 years ago

These are the eth0 NIC traffic data of a Linux device. I guess the reason for so much data is because I set the device scan time of this linux device to 30 seconds.

The xml file is large, I uploaded it to Google Drive. Please check, thank you! https://drive.google.com/open?id=17GzQydU9osGaSGYfAVQcwNBJV_EAHpia

neuralfraud commented 5 years ago

Just pushed a new build, ditched the XML junk, can you try that new version? it's in the master branch.

forever765 commented 5 years ago

I tried a new version of the master branch. When I clicked on the Query Inspector, chrome no longer crashes, even getting 10,000 objects is no problem. But what makes me strange is that although the query inspector of metrics has data, there is no data on the panel, and all the chart panels of all the dashboards are like this. picture link : https://drive.google.com/open?id=1yVurPK_4Fv-yfLPHeNx4eWxNw9FAsV-z

sorry...I mistakenly closed the issuse.

olsonnn commented 5 years ago

CPU usage on my grafana test instance dropped a lot after this build. Good job!

forever765 commented 5 years ago

After the plugin is updated to the latest version, the datasource cannot be added. It does not work according to the original configuration. The prompt "Unexpected error". I tried it many times, and changed the account number as well, until the PRTG prompts "overload protection".

neuralfraud commented 5 years ago

@forever765 New build.. Try it and let me know

I had no idea PRTG API performed language translation. That was kind of silly.

You may have to update your dashboard queries for any panel that is looking at a traffic/bandwidth channel (e.g. Traffic In)

you will now notice that the query builder shows the channels with the volume and speed. You can still use Regex style matches, e.g. /Traffic/ (or the Chinese equivalent) or simply select them individually.

neuralfraud commented 5 years ago

After the plugin is updated to the latest version, the datasource cannot be added. It does not work according to the original configuration. The prompt "Unexpected error". I tried it many times, and changed the account number as well, until the PRTG prompts "overload protection".

This may have been caused by me trying to use the secure storage for the password - i have reverted that change since it was causing problems and it is poorly documented.

forever765 commented 5 years ago

@neuralfraud Thanks for your help, I tested it with the latest version of the plugin and it worked!

Latest test situation: The first time you open the dashboard, the traffic panel loading takes about 10s (unlike before, the new version of multiple panels is almost loaded at the same time), grafana performs the dashboard auto refresh, the panel only needs 1~2s to load it! This is a huge leap!

But when I tested the addition of the prtg data source again, grafana only prompted "datasource updated" and did not display "PRTG API version: 19.4.52.3515+". This new datasource is indeed unavailable. But fortunately, the datasource built by the old version of the plugin can still work normally.

neuralfraud commented 4 years ago

Any update? Not aware of any issues, let me know if this can be closed.

forever765 commented 4 years ago

@neuralfraud Hi there, the current situation is that the first time you load a panel, it is very slow and requires 15~20s (network traffic graph). This situation is especially noticeable when using the grafana variable and refreshing the page. Can you solve this problem?

neuralfraud commented 4 years ago

That simply shouldnt happen. Whats the network condition between grafana and prtg?

Sent from my iPhone

On Nov 17, 2019, at 6:32 PM, 一位不願透露姓名的小学鷄 notifications@github.com wrote:

@neuralfraud Hi there, the current situation is that the first time you load a panel, it is very slow and requires 15~20s (network traffic graph). This situation is especially noticeable when using the grafana variable and refreshing the page. Can you solve this problem?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

forever765 commented 4 years ago

@neuralfraud Between the prtg and grafana machines is a 1000Mbps intranet, on the same switch. In addition, this slow query will only happen on the graph panel.

olsonnn commented 4 years ago

i've a question regarding this: are requests going to prtg serial or in parallel? Is there a away to change/ tweak settings here?

alotufo commented 4 years ago

I just came here to say that we've been using this plugin for several months and it has been great, but painfully slow with data requests. I thought that since the plugin hadn't seen much development that I wasn't going to see any improvement.

I tried 4.0.4 a few days ago and it has made the load times 10x faster! I tried using Grafana with this plugin on a RaspberryPi previously and it was impossible on 4.0.3. It just froze up the browser from requests. It now runs fine as well.

I only needed to change some settings for channel names and everything else worked with my existing dashboards.

Thank you!