Improve startup time

[j8r]

I've a lots of games (>100 000), put in multiple collections (~30). On my Ryzen system, Pegasus launch in ~1 minute. On a slower two core Zacate APU (1.4GHz), it goes up to more than 5 minutes. I suspect on a Raspberry PIs it would be the same. This numbers are from a "cold boot" computes, first launch of Pegasus in a session.

Observations

• in both cases, the Flatpak version has been used
• thanks to iotop, 90-100% of IO was used for the pegasus-fe process
• after a long first launch, subsequent launches are 4x faster, suggesting a cache being used. Still, the load takes a minute for the APU system, and 15 seconds for the Ryzen one.

Suggestions

I think this load time can be improved, for both systems. I don't know how Pegasus is designed internally, here are some options/thoughts:

• in my opinion, Pegasus should show the collections as early as possible, even though games are not finished to be loaded. This will allows us to select a collection when the games are still loading.
• nowadays, all end-user CPUs are at least 2 cores: it can be worth it make Pegasus multi-threaded.
• Pegasus could use async programming in order to better use the computer's recourses.
• caching can also be evaluated, or improved. I don't think it's that frequent to add/delete games, especially in multiple collections. The caching could be done per collection, and if it changed between the last time, rebuild the cache.

For the cache, you should have more insights than me. It could be an issue with the app itself, or particular to Flatpak.

[j8r]

I removed the cache rm -r ~/.var/app/org.pegasus_frontend.Pegasus/cache, but didn't changed anything. I haven't found any other location. It may be the OS doing a filesystem cache.

[mmatyas]

Yes, as you've observed, loading is an IO-heavy operation. Here's what happens:

• For each game directory, Pegasus looks for metadata files in it
• Reads each metadata file
• Checks all file entries whether the file actually exists
• Traverses the media directory, if exists, and tries to assign each file to a game

For example, assuming 100.000 games with 100.000 game files, and each game having only but a box art, that's indeed at least 200.000 IO calls. After the first start, the disk buffer, and optionally anything the OS caches, should have a lot of the files in memory, which speeds up these calls considerably. As you can see, the bottleneck is not the CPU, but the IO performance.

There have been a request for an option disabling the file existence check, and that sounds reasonable. It might also be possible to implement some manual caching too, but in practice with file system differences and special cases (eg. symlinks), I wonder how well it would actually work.

[j8r]

True. And I've not a lot of metadata files, so the bottleneck is not because of reading files.

What could be done is to list all files in directories, then do the operations accordingly. Then, it won't be necessary to check file existence one file at a time.

In the code, the file system will be represented in a structured way, as in-memory objects, where we will be able to find attributes (files) efficiently. Then, eventually, in the future, caching could be done easier by exporting this.

How I see the improvement:

1. Load all collections and games
2. Show the main interface
3. Continue to load assets in the background

Assets/images are good-to-have, but not necessary to show and have a functional UI. Also, assets could be lazily loaded.

[mmatyas]

What could be done is to list all files in directories, then do the operations accordingly. Then, it won't be necessary to check file existence one file at a time.

Yes, this could be done, though at the price of increased memory consumption. I have some ideas for making this work, but I'll have to do some benchmarking.

How I see the improvement:

By the time the theme is loaded, the list of all available assets should be available, because reevaluating the UI bindings has a performance cost. On the other hand, because the list of assets is known before running Pegasus, a smart program could be written that updates your metadata file based on your files. You could run this tool only when you know that the assets have changed, saving time for Pegasus.

Also, assets could be lazily loaded

Yes, assets are loaded when the theme loads them, but Pegasus have to traverse the media dir first so that themes have the list of assets they can load (see above). Processing the media dir file list happens at the loading part of Pegasus, but the actual reading of the asset file to the memory/GPU happens on theme access (ie. unlike game files, assets files are not checked individually for existence – a similar approach could be used for games too).

[j8r]

By the time the theme is loaded, the list of all available assets should be available, because reevaluating the UI bindings has a performance cost.

I agree, but I don't think any theme shows all games from all collections at the same time? The first collection could be fully loaded, then each collection at a time. Maybe it is impossible, no desirable to be done.

increased memory consumption

Really, that's fine - if reasonable and temporary. If one launch Pegasus, it is planned to run a video game, which will likely use at least or more memory than Pegasus. The only use case I see it to use Pegasus as a media center to watch movies, but then to use that much memory it will be required to have enough of them - which is unlikely on a low-end, low memory device.

[mmatyas]

I agree, but I don't think any theme shows all games from all collections at the same time?

They can, and they do; for example search functions do make use this. There's pretty much no limitation to how many collections you want to show at the same screen.

increased memory consumption

I think I can make it reasonable, especially relative to desktop setups; I'm more concerned about Pi Zeros and Android stuff. The OS can be nasty there.

[HVR88]

Using Pegasus' Launchbox support, my i7 system with Windows 10 was already taking over 1 minute with around 9000 titles. The library has subsequently been expanded to about 27000 titles, so I will report back shortly with a new and more detailed benchmark for startup time. All game files are on relatively fast NAS, accessed via SMB - media files locally on SSD.

All platforms (but not all titles yet) have basic media files, including box front, back, screenshot, screenshot title, cart/disc clear logo (wheel) and fanart background.

[HVR88]

Launchbox collection: 27200 titles

Pegasus initial startup: (Minutes:seconds) 8:35 until Pegasus progress bar disappears and screen blanks to start UI drawing. 9:10 when GameOS UI populated with media.

Subsequent startup: 7:40 to screen blank and 8:05 until GameOS UI.

[j8r]

Wow that's quite long. The SMB shared storage will certainly add latency, which lengthen the load time. It's IO throughput is inferior to a local SSD, in addition to be a hard drive (I guess?).

[HVR88]

Many HD (Unraid) over SMB. Sustained on larger files is 100MB/s+ but seek operations and small files are expensive. I'm really hopeful we can get an option to prevent verifying game/ROM files so that the NAS in my case doesn't need to be accessed at all during startup. In an ideal setting, I'd love to see startup time under 30 seconds total - Launchbox itself starts up in 15sec or so. And to be very clear, I generally consider Launchbox to be slow and inefficient.

I'm willing to archive and upload my entire LaunchBox folder with all media assets and XML intact for testing (it's everything except the emulators and ROMs)

[j8r]

Actually there are many calls done to detect file presence, one at a time. The bandwidth is not very relevant, latency is important in this case because time adds up for each operation.

If there is a batch call to get all files at once from a directory, it will help a lot - hence my idea of in-memory Virtual FileSystem. It can be done in other ways, too. @mmatyas has also his idea. In any case your NAS will still be accessed, but a lot less.

Indeed, what type of hard drive, local or remote, is transparent for Pegasus.

[j8r]

I tried this approach suggested by @mmatyas

On the other hand, because the list of assets is known before running Pegasus, a smart program could be written that updates your metadata file based on your files. You could run this tool only when you know that the assets have changed, saving time for Pegasus.

Unfortunately, I haven't seen any significant improvement in the startup time. Pegasus now spends more time in the Metafiles phase, instead of the Media time.

After counting more precisely:

• with the /media directories, the startup time is more or less 8 minutes.
• I've ~65000 games after counting more precisely, ~40000 of them have assets

I tried to set the assets (pointing to a assets/ directory) in metafiles instead of media/:

• having ~40000 metafiles: slow (~10 minutes)
• a metafile per collection: still slow (< 10 minutes, seems a bit faster)
• having the ~40000 metafiles merged into a single file: memory exhaustion. Maybe a memory leak, but likely due to parsing the whole file at one time - a pull parser reading a char a at time can solve this.

I'm out of idea to improve the startup time with the current state of Pegasus :disappointed:

[mmatyas]

Is the metadata file still in the directory of the games, or is it in a separate directory? (See https://github.com/mmatyas/pegasus-frontend/issues/858#issuecomment-864605534)

[j8r]

Metafiles and actual files are in separate directories. In the metafile, I specify the file location, the game, the collection and for my test the assets.

I generally put metafiles together, seperated from the games to avoid to set 30 game dirs, and for clarity purposes - easier to find them.

[HVR88]

I finally had the opportunity to test on a different platform, Raspberry Pi 4 (standard clock). Using again over 27000 ROMs/titles as on my Windows system, this time with Emulation Station metadata.

System Summary: Raspberry Pi 4B 4GB Boot Media: 1TB SSD on USB3 bus (RetroFlag NESPi4 case) RetroPie image booting to Pegasus on default theme, loading Emulation Station metadata ROMs/Titles: over 27000, mixed platforms, all on boot SSD - no large CHD/ISO images Art/Assets: videos, covers, screenshots, clear logos...

Startup time under 1 minute.

Impressive. I'd like to get my Windows setup with ROMs on the LAN starting up this quickly. Performance within the Pegasus UI is much much faster on the Windows system - it's quite laggy on the RPi4 which is a shame.

[j8r]

Yes, it really depends on the IO throughput of the motherboard itself, not really the CPU nor the hard drive. I was surprised that on my Lenovo Ideacentre 300S, with a Intel n3050, the launch time was under 3 minutes.

I noticed the UI responsiveness depends more on the CPU and hard drive, perhaps to load and for resize the images.

[HVR88]

My Windows system outclasses this RPi4 in every metric, including general IO and memory throughput. There's something else at play here and it's specific to Pegasus.

I plan a few more tests, like SSD-only Launchbox and ES-media on Windows, but it's a problem finding the time to do the setup work, like building new ES media lists for comparison. I also only have one 1TB SSD that I don't want to re-deploy. So I have to try and make a high-count ROM collection on smaller 500GB SSD for testing in Windows.

But I'm committed to this because I don't think I'll ever finish my front end setup if I can't use Pegasus all the time. There's nothing else out there right now that it's 100% terrible, whether it be Launchbox/Big Box or Emulation Station - everything else just really really stinks.

[j8r]

Not sure. Accessing through a remote storage (as you said previously in your Windows system) adds latency, which can't be compared with a local drive.

All game files are on relatively fast NAS, accessed via SMB - media files locally on SSD.

[HVR88]

This is precisely what Pegasus needs to address - where game files are stored needs to be irrelevant when starting up. I don't want Pegasus to look at my ROMs at all when starting up. Only the metadata/media files necessary to present the UI.

This is why Launchbox starts up in less than 15 seconds with my setup - it accesses its cached art and reads its dat files, but doesn't look at the ROMs at all.

I'd previously seen startup times over 1 minute with Pegasus with everything on SSD, including ROMs, with most everything on SSD (I must have had at least one patform's ROM files on the LAN) in Windows for a 9000 title collection. Even without any changes to the way Pegasus works, that doesn't compare favorably to the Pi4 setup.

I should have startup times this week for 27000 titles all from SSD on Windows. Along with timings, I should have a portable and uploadable test folder hierarchy with all the ROM files replaced by fake/empty files. If this works out, I'll create an archive and put it up for download so anyone else can run such tests.

[HVR88]

SUCCESS (in testing).

In the case of the differences in startup times I've observed, it's definitely caused by Pegasus checking the ROM files. If this one issue is resolved, I think I can switch over to Pegasus full-time on my main system.

With 27000 ROM files (1 byte dummy files) all stored locally on SSD within the Launchbox folder, Pegasus startup is less than 10 seconds - faster than Launchbox itself.

With those same dummy files on the LAN accessed with SMB, Pegasus takes over 8 minutes to start.

I will upload my test files tomorrow. A folder structure with dummy files for all the ROMs, plus two LaunchBox DATA folders, one with local paths and one with network path. To do AB testing, the DATA folder can be unzipped to overwrite the destination. The fake ROM archive can be unzipped inside the LaunchBox Games folder and another copy onto the root of a network share which should be mapped as drive "F" (a search and replace within LB's Data folder can change this to anything else as needed).

[j8r]

Thanks @HVR88 , but @mmatyas is already aware of what's causing the bottleneck (and I've also an idea). It is related to how Pegasus make his game library from the filesystem.

See his comment https://github.com/mmatyas/pegasus-frontend/issues/839#issuecomment-842191759 for the explanations.

The real deal ia not identifying the issue, but modifying the code to improve the situation, which is easier said than done.

[mmatyas]

Well it never hurts to have accurate numbers to back a theory. Thanks for testing!

[HVR88]

I was aware of the comment and the IO, the majority of which is in dealing with all the metadata. I specifically wanted to verify and confirm that the startup delays were attributed to the third bullet in the IO list - the checking of the game files existence.

• For each game directory, Pegasus looks for metadata files in it
• Reads each metadata file
• Checks all file entries whether the file actually exists
• Traverses the media directory, if exists, and tries to assign each file to a game

While I haven't looked at the code, @mmatyas descriptions indicate that the tables/lists internally are built from the metadata (in my case Launchbox's XML files) and not by indexing the ROM files themselves.

Other similar apps I've used don't do anything like #3. They don't ever know that a referenced file is missing unless a specific function is used later to check or when performing an action on such a file (like launching/playing a ROM). Apps ranging from other front ends like LaunchBox and ES, to apps like iTunes. A database, whether it be relational or simply XML files, is the only thing used at startup in such apps to build its UI. Verifying the media has benefits, like being able to hide missing entries or pop up warnings, but it's not appropriate to do so at app launch/startup whenever dealing with the possibility of large data sets and that's why no one else does it.

While this is crippling on a LAN, it will likely introduce a significant bottleneck even on a simple spinning HDD.

[HVR88]

This issue was fist mentioned here: https://github.com/mmatyas/pegasus-frontend/issues/825

Yes, for non-Steam LB games the paths are checked too, though that may or may not be the reason for a slowdown. If someone shares a huge LB setup, I'm happy to measure loading times with more detail.

[HVR88]

Testing files: http://salumba.com/LaunchBoxTesting.7z (~17MB)

Fakegames archive containing platform folders with over 50000 single-byte files named like valid ROMs Data LOCAL archive for relative/local ROM reference (inside Games folder within LaunchBox folder) Data LAN archive for absolute ROM reference on mapped drive F (inside the folder Fakegames at the root of the drive)

Data folders reference over 27000 files which will appear in LaunchBox as active games/ROMs

Required for testing:

• Working LaunchBox installation on any local drive
• Set Pegasus to use LaunchBox metadata
• Copy the contents of the Fakegames archive (not including top-level folder) into the Games folder inside the LaunchBox folder
• Copy the contents of the Fakegames archive (including its top-level folder) onto the root of a network volume
• Map the network volume to Drive F
• Copy the contents of one of the Data archives into the LaunchBox folder to test that ROM path
• Copy the contents of the other Data archive to test the other ROM path

[mmatyas]

I've just added an option to the Settings menu for skipping the file checks. Could you give it a try and see if it helps?

[j8r]

When I set verify files to false, it always came back to enabled when we go to the settings through the UI, so to be sure I have set it to true in the configuration file. Except that, I haven't noticed any significant change with and without the feature, though the loading bar improvement progression is welcome, thanks! All my games are in a separate hard drive, I've un-mount and re-mounted it to flush the cache for each test. The total load time according to the logs are for both ~65 seconds.

[mmatyas]

Hm interesting, will look into the UI bugs tomorrow. Meanwhile, make sure it is turned off, or false in the file, and don't open the menu after setting it (seems it gets overwritten for some reason). If you have console output, take a look at that too: you should get a message for each metadata file, so you can see whether they take most of the loading time, or there's something after all of them are read.

[j8r]

Reading the metadata files is quite fast, like 15 seconds. Then, there is this 50 seconds load time remaining.

I've tried again, no noticeable change for me.

[HVR88]

I'll take a look with my test files on the weekend and report back results.

[HVR88]

Initial testing is positive.

With the continuous build from Sep-17-2021 and the ROM scanning option OFF, startup with LAN-based ROM files looks like it takes exactly the same amount of time as LOCAL SSD-Based ROM files.

If you'd like any LOG/DEBUG data, let me know. I'll do some timing tests next to get a better idea of exactly how many seconds we're looking at for startup.

With regards to what @j8r wrote about a 50-second period after metadata, it would be useful to have some additional debug output to profile what's going on in that case - I haven't looked at the log created by the current version yet to see if there's anything in there for that.

[HVR88]

Timing with the test data set I posted above. Pegasus running on Windows 10 with LaunchBox scanning option - 27063 Games

LAN ROM files

Scan ROMS TRUE (default behavior): Pegasus versions 15-14 up to 15-67 (Sep-18-2021) - 5-6 minutes

Scan ROMS FALSE (new behavior): Pegasus 15-67 (Sep-18-2021) - under 19 seconds

LOCAL ROM files

Scan ROMS TRUE (default behavior): Pegasus versions 15-14 up to 15-67 (Sep-18-2021) - 19-21 seconds

Scan ROMS FALSE (new behavior): Pegasus 15-67 (Sep-18-2021) - under 19 seconds

Timings are different than when I ran the same tests back in July, which I can't explain. I'm testing through Microsoft Remote Desktop and as far as I recall, I haven't made any changes to the Windows PC running Pegasus.

The tests seem to confirm that scanning the ROM files over LAN is responsible for the extended startup time, while apparently making very little appreciable difference for those stored locally on SSD.

[j8r]

I got this script reproducing the issue, with a "cold" hard drive (freshly mounted). verify-files is set to false. media discovery is definitely what's causing the slow down. Removing the directory, or removing its logic in the script, makes loading Pegasus a lot faster.

CONF_DIR=~/.var/app/org.pegasus_frontend.Pegasus/config/pegasus-frontend
GAMES=30000

cat > test.metadata.pegasus.txt <<E
collection: test
extensions: txt
launch: echo {file.path}
directory: .
E

mkdir -p media
i=0

while [ $i -lt $GAMES ]; do
  printf "$RANDOM\r\n\e[1K"
  touch game-$i.txt
  mkdir -p media/game-$i
  for f in box_front.png screenshot.png titlescreen.png; do
    touch media/game-$i/$f
  done
  i=$(( $i + 1 ))
done

Side note: I noticed --portable is not working when using the Flatpak version, due the grep & echo commands.

[mmatyas]

Ah, then it's not finding the games that's slow, but finding the assets for the games automatically – you can find them measured separately in the log file (Metafiles and Media). Looking for assets does involve traversing the media directory; that's probably where your 50s comes from, but we can make sure if you post a log file.

[j8r]

I have:

Metafiles: Finished searching in 13983ms
Media: Finished searching in 70743ms
Game list post-processing took 10771ms

I may also try again https://github.com/mmatyas/pegasus-frontend/issues/839#issuecomment-864571100 with verify-files to false.

[HVR88]

Comparing above timings to LaunchBox source (scan ROMs FALSE) - in this case approximately 18 seconds startup:

2021-09-19T13:24:40 [i] Pegasus alpha15-67-g54e9c7da (2021-09-17) ... 2021-09-19T13:24:40 [i] Metafiles: No metadata files found 2021-09-19T13:24:40 [i] Metafiles: Finished searching in 0ms 2021-09-19T13:24:40 [i] Media: Finished searching in 0ms ... 2021-09-19T13:24:57 [i] LaunchBox: Finished searching in 16160ms 2021-09-19T13:24:57 [i] Favorites: Finished searching in 0ms 2021-09-19T13:24:57 [i] Playtime: Finished searching in 0ms 2021-09-19T13:24:58 [i] Game list post-processing took 983ms 2021-09-19T13:24:58 [i] 27063 games found

[HVR88]

I'll try to get similar startup details for Emulation Station data source for comparison as well if it's helpful. On my Pi system as mentioned previously, startup was generally very quick, even with a similar amount of ROMs as on my Windows system.

[HVR88]

Timings on Raspberry Pi4 - EmulationStation source with a few platforms using Pegasus metafiles as a test.

2021-09-20T14:30:23 [i] Pegasus alpha15-67-g54e9c7da-dirty (2021-09-17) ... 2021-09-20T14:30:24 [i] Metafiles: Finished searching in 1178ms 2021-09-20T14:30:25 [i] Media: Finished searching in 352ms ... 2021-09-20T14:30:35 [i] EmulationStation: Finished searching in 10045ms 2021-09-20T14:30:35 [i] Favorites: Finished searching in 0ms 2021-09-20T14:30:35 [i] Playtime: Finished searching in 11ms ... 2021-09-20T14:30:37 [i] Game list post-processing took 2448ms 2021-09-20T14:30:37 [i] 24178 games found

[HVR88]

Definitely @j8r's media scan is very long - and nothing like it even appears on my two systems using LB and ES sources - 70 extra seconds. The post-processing time might be a bit slow too as while it's about 4x longer than on my RPi (for 4x the files), it's 11x slower than on my Windows system.

Media: Finished searching in 70743ms Game list post-processing took 10771ms

[mmatyas]

Thanks for the measuring!

Yes, the "Media" refers to looking for assets in the media directories, and only that. If you don't use metadata files, or you don't have any media directories, it should finish quickly. "Game list post-processing" is essentially preparing the data to be usable by the themes. It is an entirely in-memory operation; the more games you have, the longer it takes.

I'm thinking about adding an option to disable the media searching too, that could also help.

[j8r]

Why not having a cache, per collection? Also, in this context (and even without), having a way to rebuild the cache (refresh button for example), and then see new games we added without restarting Pegasus.

[j8r]

I got an idea: we can have an option to have a cache per collection, with the invalidation based on mtime change. If it has not changed, no games should have been added or removed.

It can be a toggle feature because there are edge cases where this metadata may not be accurate. It may no play well with network shares or symlinks.

[j8r]

As an update, I noticed that using a different filesystem – XFS finstead of EXT4 – yields in my case with hard drives 2 times faster load time on the media step! The metafiles step is a bit slower though, like 20%, but having hundred of thousands of media, it's worth it. Easily one minute has been cut.

[HVR88]

@mmatyas > > I'm thinking about adding an option to disable the media searching too, that could also help.

Was this ever tested?

[j8r]

If it is removed then of course no images will be present. The simplest way to avoid media searching is not having media at all.

[HVR88]

If it is removed then of course no images will be present. The simplest way to avoid media searching is not having media at all.

I don't believe that's what was intended. I understood it as looking at the media files during startup only, not turning off the ability to load/display the media files during use. Example: If you have 500k media files, Pegasus is obviously not loading them all at startup, but as I understand it, it's going to look at the existence of all those files, causing (potentially) another big slowdown.

So this needs to be tested as @mmatyas proposed.

[mmatyas]

Yes, on load Pegasus goes through the files' names under media/ and tries to guess which game they belong to. On Windows and on the network this is a slow operation, and we saw that in the logs as well. I haven't added an option yet, but if you're up to some manual building, you can comment out this line. This turns off the automatic media searching only.

[j8r]

Just tried, of course I got not images at all for my games - as I thought. I did look at the code before saying that. Said differently, the mapping Game->Image is done as the loading step. Perhaps it could be done on the fly as needed, unsure.

[dontsnm]

Same issue with 71 collections with 31000 roms. Over 5 minutes to show the theme in android.