Complete system freeze during first scan with huge library

[SebinNyshkim]

I've built Resonance from AUR.

During initial scan of my library (39,796 files, ~288 GB) Resonance burns through all of my 16 GB of system memory while trying to populate the database. This in turn locks up the entire system, forcing me to hard reset the machine.

From #32 I gather that there is still a bit of memory management optimization needed, especially with larger media libraries with lots of high res cover art.

But what could make Resonance use this much memory during the first scan? Is it trying to keep the entire library in RAM?

[nate-xyz]

Likely due to python/mutagen. Yes it is keeping the entire library in RAM, which will need to be optimized. (Likely by importing each track individually rather than getting every track's info at once) One issue with importing individually is that it opens up the possibility for partial imports if the import process is interrupted. As it is now, the library only finishes constructing the db if full sql transaction is completed. Maybe the partial import problem is how it is and will be solved by https://github.com/nate-xyz/resonance/issues/9

I would love to multithread the import process but idk if I can do that with pyo3 due to the python GIL (https://github.com/PyO3/pyo3/issues/879), and last time I tried multithreading it within python there weren't any performance gains.

[SebinNyshkim]

I can see where you're coming from with this approach, but as other issues have shown, keeping the entire library in RAM scales very poorly once the library is very large. Resonance already uses 10GB of RAM on my system just to import the files, even before the cover art is extracted. I think you'll agree that this is excessive for a music application.

It would indeed probably be better if files were imported into the database one at a time or in several chunks of perhaps 1,000 files each until there's no more files to scan, rather than in one single huge batch. There would be the potential risk of an incomplete library because the import was interrupted, but it would make it possible for someone like me to import my music at all.

As it stands, I can't even start using and testing the application. I think that's a far bigger showstopper than the risk of partial imports. If that happened, at least some of my music would be there and I could just try importing again. Right now I'm stuck with nothing.

If interrupted imports are a concern, perhaps you could set a flag somewhere to indicate that a scan has been started, and once that scan is complete, the flag is unset. If that flag is still set the next time Resonance is started, that could cause a dialog box to appear asking the user to rescan: "The import was interrupted the last time and your library may be incomplete. Would you like to try scanning for media files again?"

Another music application that uses python/mutagen is exaile. You could have a look at how it handles media imports with lower RAM utilization while using python/mutagen.

[nate-xyz]

I can see where you're coming from with this approach, but as other issues have shown, keeping the entire library in RAM scales very poorly once the library is very large. Resonance already uses 10GB of RAM on my system just to import the files, even before the cover art is extracted. I think you'll agree that this is excessive for a music application.

It would indeed probably be better if files were imported into the database one at a time or in several chunks of perhaps 1,000 files each until there's no more files to scan, rather than in one single huge batch. There would be the potential risk of an incomplete library because the import was interrupted, but it would make it possible for someone like me to import my music at all.

As it stands, I can't even start using and testing the application. I think that's a far bigger showstopper than the risk of partial imports. If that happened, at least some of my music would be there and I could just try importing again. Right now I'm stuck with nothing.

If interrupted imports are a concern, perhaps you could set a flag somewhere to indicate that a scan has been started, and once that scan is complete, the flag is unset. If that flag is still set the next time Resonance is started, that could cause a dialog box to appear asking the user to rescan: "The import was interrupted the last time and your library may be incomplete. Would you like to try scanning for media files again?"

Another music application that uses python/mutagen is exaile. You could have a look at how it handles media imports with lower RAM utilization while using python/mutagen.

No yeah I agree, I will be changing it, it's definitely not ideal.

[legacychimera247]

need to say, my library is roughly 300 gb (flac mostly) and even if slower, first import worked

maybe you should try the flatpak version?

[SebinNyshkim]

The total size of the music files or the audio format is besides the point.

By keeping the database in RAM, Resonance consumes an absurd amount of RAM for a music player with a large number of media files. Flatpak can't change that.

A music player should not take up 10 GB of RAM scanning for files...

[nate-xyz]

I have ~1200 albums in my collection, flac, varying samplerates and bitdepth. Same symptoms here. Will not try again until total system freeze is solved. It's bad advertising for a rust app to kill your computer with memory issues.

Not sure what being written in rust has to do with it but ok lol

[secretmango]

honestly, how much RAM do you have? XD

I think this is a dupe of #32

I had the same issue, needed to hard shutdown my PC.

[SebinNyshkim]

Not a dupe. #32 the app is still operational. In this case building the database with a huge library consumes ridiculous amounts of RAM and freezes the entire system. I issued this back when I had 16 GB of RAM, I've since upgraded to 32 GB of RAM. But that is irrelevant. My point is, this shouldn't happen with any amount of RAM. Both cases might be linked to the same underlying cause. Until that is proven however we cannot say for sure. IIRC, it might be linked to a library that's used for reading tags and album art?

[legacychimera247]

Had a try right now again, used about 8 gb to scan my music library, and now on the app i'm at 10.6 gb of used memory, probably as it's fetching artist images

still that's way too much even with 16 gb available

unfortunately, despite being lovely i feel like this player has been abandoned...