Improvement: defining fresh time in real-time monitoring

[0bi-w6n-K3nobi]

Good morning, @allinurl

I didn't look for GoAccess issues forum. But, I wonder if it would be possible to set the refresh time when the browser is monitoring the real-time.

My computer at work in a little bit old. And when I monitor the larger site -- about 500 requets per second -- my Firefox loads the CPU at around 60% ~ 80%. If I use the No-Script add-ons it drops to 60%. And finally, when I switch to a new or empty tab the load drops to ~ 0%.

Would it be possible to add an option in the configuration panel such as ?

refresh time

• real time
• 3 s
• 7 s
• 1 m
• ...

I believe that this has to do with the JS present on the page.

Thanks.

[allinurl]

Great idea. I'd look into this. It may be a simple change in here.

[0bi-w6n-K3nobi]

Hi @allinurl

I may be mistaken, but I don't believe the problem is on the service/server side. I think it would be on the client/browser side ... in its acquisition rate when connected to the WS server.

Thanks.

[allinurl]

Good question. I think this can be done either on the server or the client. If done on the server, the server will push to the websocket data every X seconds. If done on the client, the server will keep pushing data at the same rate (0.8 secs) but won't re-render/refresh the content untill X seconds have elapsed.

To be honest, I feel like doing it on the server side may be more efficient to the client just so it doesn't get bombarded by data that won't be used until the last client refresh. Thoughts?

[0bi-w6n-K3nobi]

The Good, the Bad and the Ugly ...

Really, any implementation will have pros and cons.

Modifying on the server - if there are many requests [per sec] to be processed there will be load spikes.

I personally would vote against -- because my biggest site loads 50% of 1 CPU all the time. If we extend the wait time, I believe that the CPU will not be able to process in due time. This site produces 4,000 to 12,000 requests per sec. And once this waiting time has been defined, it could only be modified by restarting the program.

I believe that this solution is suitable for smaller sites. Yeah. I know that would save resources on both the client and the server. However, it would not be suitable for sites with many requests [per sec]. Or... Would it be possible to make a self-adjusting waiting time? According to the load or number of requests in the last iteration in the processing loop. Or Maybe just on the WS server?

Modifying on the client - just reduces the load on the client/browser.

There is not much to talk about here. Ideally, the client should be able to inform/setting the WS server of the update rate. Thus avoiding unnecessarily sending information to the client/browser. Therefore, the rate could be changed by the customer at any time -- via panel setting. Some resources could be saved -- at least from the WS server and the client/browser.

PS: Have you ever thought about a communication interface -- at least with the WS server -- to resetting some variable in GoAccess when it is running as a real-time Server?

I’ll talk to you soon.

[0bi-w6n-K3nobi]

Hi @allinurl

I believe that the GoAccess server/program has to process all requests regardless of the waiting time in the processing loop. The definitive solution is to change the refresh time. I believe this solution is between the WS server and the client/browser. And so this is where we have to make the adjustment.

Then we have the options below:

• reduce the sending rate of the WS server;
• only reduce the rendering rate on the client/browser;

For the implementation, we have the possibilities:

• Adjust the rate of the WS server via GOACCESS configuration -- fixed;
• Adjust the rate ... via the configuration panel on the client/browser -- adjustable;

This last item will need implement an interface with the [WS?] server.

See you then.

[allinurl]

I just pushed a commit for the server-side. Please take a look at it and let me know how it works for you. Like you said, adding this to the client-side it may be the way to go too.

I'll be pushing soon v1.4.3 to address f48efc5b5c79a387cbab72c1c700c89aa258f20e

[0bi-w6n-K3nobi]

oops ... You want mean d8a0df3 .

[allinurl]

Yes, the commit I just did for this issue is d8a0df3. v1.4.3 is going out either today or tomorrow due to f48efc5 :) Let me know if --html-refresh=<secs> works for you.

[0bi-w6n-K3nobi]

Good evening @allinurl

I did the tests on my biggest website. In fact, I thought it best to use milliseconds ... maybe the most appropriate is tenth of a second.

So, I used the --html-refresh parameter with 2, 4, 8 and 25 seconds. And even with the higher value, GOACCESS was able to process requests. But, as I imagine, the server's CPU is pulsating in bursts of processing. As should be expected.

Well, I was thinking about it. I think what I think suitable for the server would be a waiting time [or sleep time] between each request. This would smooth out the CPU usage on the server. But, then the data transmission via the WS server would still be after the processing of each LOG or all LOGs, in the real-time [ tail ] processing LOOP. Or maybe after N passes through the real-time processing LOOP, then the WS server would send the data to the client/browser.

What do you think of this?

Of course, this time would be short, as it would be between each request. For eexamplexample. example: if the server's CPU processed 4,000 requests per second, this delay would be:

delay = (website requests per second) / 4,000

Another way would be to self-adjust. Assessing how much the CPU is capable of processing and how many records are left to be processed. So the calculation would be dynamic.

This would ease the CPU load, without compromising or delaying the processing of requests.

Well, that's just a rough draft on the subject. I would like to keep an open topic about it.

Thanks.