search

wizzomafizzo / mrext

Collection of extensions and utilities for the MiSTer system.
GNU General Public License v3.0
173 stars 11 forks source link

Indexing a large number of games kills process #24

Open wizzomafizzo opened 1 year ago

wizzomafizzo commented 1 year ago

When indexing an extremely large number of games on a system, app will exceed the amount of storage available in /tmp (ram) and get the process killed. This issue has been reported specifically with launchsync

jamescadd commented 1 year ago

Is there a workaround?

wizzomafizzo commented 1 year ago

Not yet. I will take another look but I was not able to reproduce the issue reported yet

aryko1872 commented 11 months ago

If it helps you reproduce, I'm seeing this issue using this sync file: https://raw.githubusercontent.com/aryko1872/mrext-launchsync-syncfiles/main/Arcade-15Khz-Horizontal.sync

jamescadd commented 7 months ago

I can repro this issue 100% of the time, is there anything I can do to help get it fixed? Maybe the db provider can be figured to automatically flush to disk, or to skip creating indexes until the entire db is populated, or something else easy to prevent the memory issue?

Alternatively, I would like to create the database file elsewhere (probably on RetroNAS) and drop it somewhere that the extensions like search would pick it up. Is it possible to move the database file to a share-mappable location like /media/fat/games and document the schema & provider needed for this scenario? I suppose there would also need to be a setting like "manual indexing" so that the indexer doesn't overwrite the file. Just an idea, open to anything that would fix the search REST API.

wizzomafizzo commented 7 months ago

I was still not able to reproduce this. But my best guess is that the indexing process is filling up the RAM with data. It's very basic, it just dumps a giant list of filenames to /tmp (in RAM) and then when the process is complete it copies that file to the SD card. The "db" file is just a gzip of text files splitting the filename lists up by system name.

I don't want to change it do straight to SD because this will dramatically slow down the process for everyone else. Maybe it could write to SD in chunks, say after each system, and clear out the /tmp usage each finished system. I don't know how much work that would be though, it could be pretty significant

launchsync doesn't even use a stored search.db file. It creates it on the fly each time it's run. So I'm not sure an option to specify its location will be helpful in this case. But I can look at adding it as an option

I'm not at home at the moment but I can revisit this in a couple weeks. I'm sorry it's been causing you all this trouble, it's a pretty tricky one to work out

jamescadd commented 7 months ago

No trouble at all, and thanks for the reply! Just a thought, but I've been wondering if there's another way to solve the problem of finding the file name to launch that would sidestep the issue of indexing.

For example, give people the option in settings that says "Enable HTGDB" and then assume their roms are at the locations provided in https://github.com/frederic-mahe/Hardware-Target-Game-Database (extensions can then download the dbs from the repo and search them instead).

Another approach might be to let people provide their own SMDB files, for example RetroNAS packs in SabreTools which can scan your library and create a SMDB of the files. People could upload their own in the web interface and bypass the built in indexing. I still do feel having RetroNAS perform indexing would be a valuable feature given its popularity, it seems like a thing Ansible can do though I'm not familiar with it.

wizzomafizzo commented 6 months ago

I finally fixed it launchsync.zip

thiagoauler commented 6 months ago

That’s good news, Wizzo!! Thanks!! One question: This fix is available through update all, or for now we should manually install the zip provided??

wizzomafizzo commented 6 months ago

That’s good news, Wizzo!! Thanks!! One question: This fix is available through update all, or for now we should manually install the zip provided??

Right now use that zip but it will be fixed in the next released version too!

aryko1872 commented 6 months ago

Thanks for the update!

Adam Rykowski @.***

On Sat, Mar 9, 2024 at 9:12 AM Callan Barrett @.***> wrote:

That’s good news, Wizzo!! Thanks!! One question: This fix is available through update all, or for now we should manually install the zip provided??

Right now use that zip but it will be fixed in the next released version too!

— Reply to this email directly, view it on GitHub https://github.com/wizzomafizzo/mrext/issues/24#issuecomment-1986867971, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACSNRYWHQNV4LPU3PORQOATYXMKEDAVCNFSM6AAAAAAZKSQ5FSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWHA3DOOJXGE . You are receiving this because you commented.Message ID: @.***>

jamescadd commented 5 months ago

@wizzomafizzo any chance we can have a release sometime soon so the lazy among us can try these fixes? :)