Closed Floflobel closed 2 months ago
This has been fixed.
Please update the advanced_settings
> filter_statements
section of the config to the latest layout.
Beautiful as usual. Thank you for the responsiveness.
Np!
Also to help with any confusion between filter_statements
and behavioral_statements
.
filter_statements
under both basic_settings
and advanced_settings
affect how MUMC grabs media_items from Emby/Jellyfin.behavioral_statements
affect the media_items after they have been grabbed from Emby/Jellyfin.Hey @Floflobel I have made an attempt at shrinking the runtime (RAM) footprint with v5.8.22-beta. If you get a chance try it out and let me know if it helps condense your setup back to a single script.
It could just be you have more media_items and/or users than I can fathom; hence why a single script exhausts your RAM. Hopefully the removal of redundant and extra data in the v5.8.22-beta script was carrying helps.
Wow, that's a really nice notification I'm getting here. As soon as I have some time (in between now and Monday) I'll give it a try and get back to you. This is indeed an important point, even though I've separated my entire launch into several files. Thanks for your time, I'll get back to you as soon as possible.
I seem to be getting to an OOM very quickly even though I've only done ~20 users. What can I provide to help you?
The problem occurs during fetching of media_items, I managed to make 20 users then I have an OOM.
I am currently testing on 2356 media.
Concerning RAM: It's quite complicated to see, but I'll try to get this information. All I know is that I have 15GB of RAM available.
EDIT: I reach more than "13141.9MB" in less than 3 minutes.
EDIT: I reach more than "13141.9MB" in less than 3 minutes.
Whoa! That's a lot of RAM consumed in a short time.
The problem occurs during fetching of media_items, I managed to make 20 users then I have an OOM.
This makes sense. During the fetching of media_items MUMC has to store the information it needs for post processing later.
I am currently testing on 2356 media.
Concerning RAM: It's quite complicated to see, but I'll try to get this information. All I know is that I have 15GB of RAM available.
Thank you for the info!
I will create >=20 users and figure out a way to get up to ~2400 media_items. Then see if I get the same behavior.
Side Note: If I see the same behavior on my test server the only way I can think to further decrease the amount of data needing to be stored would be to fundamentally change how MUMC works.
MUMC_v6.x.x
Thank you for all this comprehensive information.
The question I have is what have you changed recently? I didn't have such a big problem until I updated in the last commit. I was managing with ~15GB of RAM to get through all my users and media (I'll try to have a look in the code).
For the record, with the old version I'd managed to reach my "almost" ideal setup. Everything worked and I could switch to all my libraries with separate configuration files.
Regarding the big changes you mentioned at the end, this is indeed what could solve all the problems. I thought of it when I saw how MUMC worked. But your arguments are completely valid, it's a very big change and a lot of time invested.
Thank you for all this comprehensive information.
Np!
The question I have is what have you changed recently? I didn't have such a big problem until I updated in the last commit. I was managing with ~15GB of RAM to get through all my users and media (I'll try to have a look in the code).
I am guessing my attempt to simplify the data stored during fetching actually made it worse. Previously the complete data for each media_item was stored for all behaviors (e.g. favorite, whitetag, blacktag, etc...) and for each user. Along with extra data to make post processing go a little faster.
My goal was to only save the minimum amount needed. Sounds like I made things worse and now additional memory locations for data are being created and exhausting your RAM sooner.
Regarding the big changes you mentioned at the end, this is indeed what could solve all the problems. I thought of it when I saw how MUMC worked. But your arguments are completely valid, it's a very big change and a lot of time invested.
I tried a sloppy-copy-paste for this to see if it was doable. It appears it is. I need to work thru how I want to handle the multiprocessing to parallel some/all users evaluations of the single media_item. It's also going to completely change how console data is shown when running.
It's still a big change. But not as big as I initially thought. But again, don't get your hopes up too high yet. I don't what I don't know.
@Floflobel No rush. Take a look here whenever you get a chance.
@Floflobel I was able to find the issue causing the memory to be exhausted. My own fault, was saving a list to itself. Which caused it to double in size each time it was saved.
The latest release (v5.8.29-beta) on the beta branch has the fix.
I tested with 24 users and >8000 media items. Memory usage peaked at ~200MB of RAM. YMMV.
Apologies for the slow fix on this one.
I can confirm that I'm also running at around 200mb/300mb with many more users.
I can also confirm that there is no longer any point in carrying out the work you've written about (#126).
I'm going to try with a lot more data as I've currently split everything into several configuration files.
Don't worry, it's already fantastic.
Thank again for you work!
Great! Let me know how things turn out.
Hello,
I'm looking to optimize my speed as well as my RAM when I run MUMC on a large library with lots of users.
I've noticed that the script still performs the process for "Whitelisted" media when I'd like to ignore it completely.
https://github.com/terrelsa13/MUMC/wiki/mumc_configyaml-advanced_settings-behavioral_statements-media_type-conditional_behavior-action_control
But currently it's still processing my whitelisted libraries even though I've set the parameter to disabled.
Here's my configuration: