search

ElvishArtisan / rivendell

A full-featured radio automation system targeted for use in professional broadcast and media environments
197 stars 63 forks source link

Memory consumption in rdairplay and rdpanel #941

Open lbracci opened 4 months ago

lbracci commented 4 months ago

Rivendell 4 is amazing, but I've noticed that RDAirPlay and RDPanel consume a lot of memory depending on the number of sound panels configured in RDAdmin. This didn't happened in Rivendell 2.7.0

Example:

In our production server, we have 100 panels in RDAirPlay, and the memory consumption is Virtual 4435 MB / Resident 3591 MB / Shared 59 MB / 45% (Our RAM is 8 GB, we are using Linuxmint 21 Mate)

With Rivendell 2.7.0, using 100 panels and playing a log, RDAirPlay consumes Virtual 268 MB / Resident 49 MB / Shared 29 MB / 0.6% (also with 8 GB, UbuntuStudio 18.04)

RDPanel have the same memory behavior in 4.1.3... also, when started, using 50 panels, it takes about 10 seconds to be responsive and show the buttons. When using 5 panels, it is very fast.

Could this be due to some Qt5 widget consuming too much memory?

Putting that aside, Rivendell is great and the version 4 has been working pretty well for us.

Cheers.

alexolivan commented 4 months ago

Interesting... I'm for a long time investigating a pair of memory Leaks hidding somewhere in both ripcd and caed Rivendell processes, so anything memory consumption related gets inmediatelly my attention.

Are your memory figures gathered, just after a fresh Rivendell startup? or on a Machine that was running for some time?

Cheers.

lbracci commented 4 months ago

Interesting... I'm for a long time investigating a pair of memory Leaks hidding somewhere in both ripcd and caed Rivendell processes, so anything memory consumption related gets inmediatelly my attention.

Are your memory figures gathered, just after a fresh Rivendell startup? or on a Machine that was running for some time?

Both cases. I've tested it in a fresh install with no carts added, and also in my production server with 26,000 carts and 100 sound panels.

EDIT: To reproduce the problem, just configure RDAdmin->Manage Hosts->RdairPlay-> and set 50 user panels. Then run RdAirPlay, and check the memory consumption.

lbracci commented 4 months ago

Our problems with RDAirPlay aren't only with memory; when the user press the button "ADD" to add a new cart into the button panel or into the logs, there is a delay of 2 or 3 seconds after choosing the cart, until the buttons get yellow and the user can click to place the cart. Also, there is another delay of 2 or 3 seconds after the user chooses where to place the cart, until the cart is placed (this occurs when we choose 50 user panels for RDAirPlay; we need this because we have ~35 live radio shows in the week, and each one need to have their own button panel, with their beds, intro, outro, FXs, and other audios).

I've found a solution: changing the constants PANEL_MAX_BUTTON_COLUMNS and PANEL_MAX_BUTTON_ROWS in lib/rdbutton_panel.h, and recompile. Both constants defines the maximum of buttons in a panel: 40x23 buttons.

There are some procedures in rdbutton_panel.cpp and rdsound_panel.cpp that are executed in each button of the panels, for each row and every column, from zero to PANEL_MAX_BUTTON_COLUMNS, and from zero to PANEL_MAX_BUTTON_ROWS. If this must go through 50 panels, this will run through 46,000 buttons! (40x23x50). In previous versions of Rivendell (2.7.0 and others), with only 5x5 buttons, there was a maximum of 1,250 buttons.

I understand there is a need of a maximum of 40 x 23 buttons for people running RDAirPlay maximized in a 4K monitor, but we, at our radio station, don't need so many buttons in RDAirPlay, not in RDPanel. I've changed the values of both constants to ones more reasonable for our needs (PANEL_MAX_BUTTON_COLUMNS=9 and PANEL_MAX_BUTTON_ROWS=8), and now RDAirPlay occupies much less memory with 50 user panels (resident: 338 MB/virtual: 997 MB). The maximum of buttons is 3,600, something more reasonable for our hardware. Also, the delay of 2 or 3 seconds when adding carts dissapeared!

I would like to recommend, if possible, that the values of PANEL_MAX_BUTTON_COLUMNS and PANEL_MAX_BUTTON_ROWS can be set in /etc/rd.conf, so RDAirPlay does not have to be compiled when those values need to be changed.

ltyndale commented 4 months ago

This has actually helped with the same issue that I've been seeing on my Raspberry PI build of Rivendell 4.x. Memory use has been very high with rdairplay and when starting up it takes around 20 seconds before it is useable and it is much more sluggish to respond then Rivendell V3. On a PI I expect things to take a little longer, but while V3 was a little sluggish but acceptable on a PI, the additional resources that RDAirplay seemed to require on Rivendell 4 made it unreasonable to run on a PI for anything other then testing.

Reducing these 2 settings and recompiling has actually made rdairplay run the way it did on Rivendell V3 again. Memory usage is what it previously was and rdairplay runs at about the same responsiveness as it previously did.

I'd agree that if this can become a setting in rd.conf or rdadmin that would be great. I can understand that with the windows that can be resized now you'd want to be able to fill the window with buttons. But the downside of that is the additional memory and resource usage.

ElvishArtisan commented 4 months ago

There are some procedures in rdbutton_panel.cpp and rdsound_panel.cpp that are executed in each button of the panels, for each row and every column, from zero to PANEL_MAX_BUTTON_COLUMNS, and from zero to PANEL_MAX_BUTTON_ROWS. If this must go through 50 panels, this will run through 46,000 buttons! (40x23x50). In previous versions of Rivendell (2.7.0 and others), with only 5x5 buttons, there was a maximum of 1,250 buttons.

I understand there is a need of a maximum of 40 x 23 buttons for people running RDAirPlay maximized in a 4K monitor, but we, at our radio station, don't need so many buttons in RDAirPlay, not in RDPanel.

Good catch! That is exactly the design logic going on here. While changing the COLUMNS / ROWS constants solves the issue for your case, I'd rather explore a way to make these settings more dynamic --e.g. tune themselves automatically on the basis of what is actually needed based on the current rdairplay(1) geometry-- rather than having the user manually tune internal code configuration. One of the things that make this challenging is that it is possible to trigger soundpanel buttons even when that are not visible in the UI (via RML). Thus, we cannot dynamically create/destroy them on the basis of their visibility state alone, but require the underlying play-out logic to be resident in memory and ready to respond to RML commands at all times.

Even as it is, the default hard-coded button 40 x 23 limit is a hack for reasons other than the performance drawbacks (suppose the user has maximized rdairplay(1) on an 8k display, or on two 4k displays in an XCINERAMA config, or <name-your-outlier-scenario-here>). Perhaps we need a hard limit on the number of buttons in a panel, and then resize the buttons themselves when application window is resized?

ltyndale commented 4 months ago

I'm not sure if this would be possible, but what about an rdiarplay setting in rdadmin to allow each user to determine what would be best for their situation for columns and rows, similar to how you can specify how many system and user button panels you have. Then let it be up to the user. Those with resource or memory constrained systems and running on lower resolution screens can select a setting that'll consume less memory. Those who want more buttons available can set the numbers higher. I don't know if this would be a possibility or if it would cause database issues though.