search

RetroPie / EmulationStation

A Fork of Emulation Station for RetroPie. Emulation Station is a flexible emulator front-end supporting keyboardless navigation and custom system themes.
Other
856 stars 344 forks source link

Slow initial startup shows static ES title screen for >3 minutes with no progress indicator. #457

Open rsn8887 opened 6 years ago

rsn8887 commented 6 years ago

I have many thousands of roms present, some of them without game list files. In this situation, ES takes forever to load on power up (~3 minutes). I think it is looping over all the roms to find the filenames. I don't understand why this is so slow, since it doesn't have to scrape or unzip the roms, and ES is written in C.

It would be nice to a) speed up the launch of ES on boot, b) display some kind of progress or at least a number in the bottom left corner, such as "6,125 roms found" with a number that increases.

treymerkley commented 6 years ago

From what I can tell, there were plans to fix this in upstream. Whether or not that's happening on the fork, no idea.

https://github.com/Aloshi/EmulationStation/issues/191

aarononeal commented 4 years ago

I added a comment in the upstream describing the problem and solution. It's not XML vs. SQLite.

pjft commented 4 years ago

@aarononeal upstream isn't maintained anymore, as far as I'm aware, but if you'd want to submit a PR here we'll gladly review it. Thanks!

EDIT: to be clear, I was referring to the changes in the file handling, as the progress handling you added there seems similar to the one we have here, unless I'm missing something.

rsn8887 commented 4 years ago

The proposed fix mentions that the user should turn ON "Parse Gamelists Only". If I do that, my boot time is already very short. Of course it is short, because Emulationstation just parses a few XML files then.

My issue pertains to "Parse Gamelists Only" being OFF, which is the default option, and which is the only option that displays all my roms including subfolders correctly. I don't want to turn on "Parse Gamelists Only". I want Emulationstation to find and display new roms, and to display roms in sub folders correctly with subfolders that can be browsed as sub-menus. For that reason, I kept "Parse Gamelists Only" off.

I still don't understand why "Parse Gamelists Only" only should be required to be ON for fast boot time. Again: The app doesn't have to unzip or scrape roms, it just needs to go through the directories to find new rom files. That should take seconds, not minutes, even with 6000 roms. Reading a folder with 6000 file names and displaying the names takes less than a second in the shell.

Even my Vita with PFBA emulator can scan 13,000 roms in a few seconds on boot up of that emulator.

aarononeal commented 4 years ago

Of course it is short, because Emulationstation just parses a few XML files then.

That's not true. Without this fix, it checks the filesystem for every entry in the XML. This is extremely slow over high latency networks or media.

rsn8887 commented 4 years ago

Ok maybe it is slow for you, but for me "Parse Gamelists Only" was always quite fast. Maybe 10-20 seconds boot time. But not minutes. The problem was when that setting was OFF.

aarononeal commented 4 years ago

You didn't mention the media you're loading from.

rsn8887 commented 4 years ago

USB Stick.

rsn8887 commented 4 years ago

The problem with the setting being ON is that only correctly scraped roms will be displayed. For example the Amiga emulator will not display any of my config files etc if that setting is ON, because well the scraper doesn't scrape them. I would still like to see them without editing xml files by hand.

aarononeal commented 4 years ago

There are 2 main paths, one parsing only game lists only and one scanning folders.

I optimized the parse only path by eliminating all stat calls to the filesystem. In my high latency test case, this had huge (160x faster) impact.

Looking at the other code path of scanning folders, I see stat is called for every entry in the gamelist multiple times. Depending on the media that's not as cheap as you might think. In some cases it results in updating last access time for each file. This may be contributing to the performance problems in that path as well.

hex007 commented 4 years ago

I have some time to spare this week. Will take a look, unless any other maintainer is interested.

hex007 commented 4 years ago

lstat is used mostly to follow symblinks for files. I am working on 2 approaches for this, either completely disable the symblink resolution flow using a param in settings or hard code the symblink resolution and work with the OS to auto follow the links. The second seems to be harder to figure out. Might take while so will keep posting status on this thread.