[Feature Proposal] Improving startup time

[chrisheib]

Actual behaviour

Ultrastar takes a looooong time to start up.

Expected behaviour

Ultrastar starts quickly.

Steps to reproduce

Set the songpath to a directory with maaany songs, in my case ~400Gb ~~ 13000 songs and start it up, for best effects on a freshly booted computer to mitigate any OS-IO-caching that might be in place.

Details

We have an external hard drive with a lot of songs. Sometimes USDX just crashes and we need to restart it, which takes sometimes close to 10 minutes. There are a lot of faulty songs in the directories, but I think USDX should be designed in a way that doesn't let those errors impact the loading speed.

I looked into the code, and saw two possible approaches to speed up the progress.

1. At first glance the song loading is designed to be threaded, but it does a lot of work right away, delaying the menu a lot. Is this intended? Could more work be put into the thread that then gets started after the menu is done loading?
2. I don't frequently change my song libraries, so i thought it would be possible to save the 'work' the thread does. My idea: Force the thread to do its songlist-work for every folder in the ini, serialize the songlist-structure, and write it to a file in the base path stated in the ini, maybe as json. Then on startup USDX could look for the songlibrary.json (or however it would be named) and if it finds it, load it up, instead of parsing the whole directory structure again.

For the first step I would implement it in a way that lets you activate this via ini and need to either run a function from options or start USDX with a specific parameter to refresh the song libraries.

As a second step it might be worth a look to save hashes of each folder (I believe there are functions for that) and look for differences with the saved library to decide wether to use or discard and refresh the library.

What are your thoughts on this? Is something like this already planned/being worked on, and, as I'm new to the project, would it fit the principles behind USDX? I guess I could code a first version of the 2nd idea in the next few days.

[bohning]

Out of curiosity, how long exactly is "a looooong time"? It takes less than 15 seconds for 3100+ songs for me...

[basisbit]

easiest workaround would probably be to use an SSD instead of a HDD.

[s09bQ5]

Related to #322

I am not sure how much of the information is already cached. There is an SQLite databases with a table that has one entry per song which is managed by src/base/UDataBase.pas. If you feel like improving the startup time yourself, your help is really appreciated.

[skerit]

I've noticed this too. Though I don't have 13.000 songs (which is an impressive amount of songs!) my instance also took a long time loading +/- 900 songs. And by "long time" I mean about 1 minute.

It kind of kills the mood should a crash occur during a party :)

I also use an NFS share to access my ultrastar profile & songs, so I'm curious: what does it do during boot exactly?

Maybe a kodi-like approach is a good idea: cache the library & load it in the background during startup.

[basisbit]

the "background-scanning" approach indeed would be the best solution. Unfortunately, multithreading is a bit complicated in objective pascal, and the last 5 years or so, no developer volunteered to do that work. In fact, there exists a working sample implementation of how this could be done in the UltraStar Play project ( https://github.com/UltraStar-Deluxe/Play/blob/master/UltraStar%20Play/Assets/src/model/SongMetaManager.cs#L47 ), but sadly there is almost no progress in that project because it is missing someone who knows how to use Unity to create UI stuff.

Anyways, back to your question: Besides UI initialization, parsing the config and the theme stuff, during startup USDX scans all the configured folders for any *.txt file, then fully parses these files (to check validity and only show valid files in game - also there is tons of code to fix bunch of song file issues on the fly), it checks for each song file if a usable audio file exists, it opens and processes the song cover image files to get their sizes, and updates the ultrastar.db accordingly.

Yes, having song folders on a NFS mount makes the bootup time much slower, because file access to the 900 song files plus the audio+video+cover files takes much longer. I suggest storing your song files on a cheap SSD directly connected to your computer (either built in or USB 3). That should reduce game start time by a lot.

[hofingermarkus]

I think most of the time using an old saved database/song-list/json/etc. from the last run - without running a new check would be really good enough.

Just think of a party with friends. Usually one prepares the songs, microphones and ultrastar in advance and tests if everything works. In this process all of the songs get scanned and checked, so the startup can take the 10-15 minutes there. However, once the party runs it would be great if ultrastar could just startup in a fast mode using the 'pre-generated' database/song-list. In this way accidently closing of ultrastar (yes that can happen on parties ;-) ) or program crashes are not as bad startup time wise.

The 'use last song-list, (skip checks/updates)' feature could be added as an optional setting that is deactivated by default. In this way nothing changes for most of the users, but an educated user is able to activate it specifically for a party. This would also keep the programming effort to a minimum.

[basisbit]

hofingermarkus , I suggest you to move your songs folder to a SSD. We won't put extra effort and time (non paid) for UltraStar Deluxe into solving issues of people using very slow harddrives. Sorry. The developers free time is rather valuable and there are much more important parts of the code that need to be worked on first.

You can get a good 500GB SSD for roughly 50€/$ and that will solve your issue.

[jschwartzenberg]

I have a system on which it takes up to 30 minutes to start UltraStar DX. Songs are loaded over a network connection. I tried both NFS and SMB, but both have this issue.

There is neither notable CPU load nor notable network load during that time. I have not been able to figure out what the bottleneck is.

[basisbit]

The bottleneck will be latency, no caching, and every file being accessed over network. Solution: don't use network storage for the song folders.

[jschwartzenberg]

No, the botttleneck is somewhere else. I just tried it on another system and there UltraStar DX starts in 5 minutes with the same folder set, also via the network.

Both systems run Ubuntu 18.04, but the slow one is an ARM device while the fast one is an Intel device. I would like to use a debugger to see what is going on. I never worked with Pascal/Delphi/Lazarus, but I guess I can just install the environment and try to debug this. If anybody here has any tips for that, I'd appreciate it.

[s09bQ5]

Try perf to see where the time is spent. You want to sample an event that also triggers while the cpu is sleeping, like cpu-clock.

Don't forget to run configure with --enable-debug to get symbol names.

[jschwartzenberg]

Thanks a lot for the hint! I ran perf record --event=cpu-clock ultrastardx and got the following: perf.data.gz

Maybe you want to take a look, I'll also see if I can manage to read this data myself :)

[jschwartzenberg]

I installed https://github.com/KDAB/hotspot and did some more analysis as it seems just a perf.data is not enough without the rest of the system available. Here is a screenshot:

I'm afraid this is still not enough info to deduce where the delays are coming from.

[basisbit]

just did some tests here, and indeed, startup is really slow when scanning through folders which are on a SMB fileshare, when the files are not already in the local file cache (might require opportunistic locking and enough free storage space in the local file cache). When doing the same on a system which is configured to use local filecache for the smb share, startup time is only about twice as long as with only local files., So yes, thats just network latency slowing down the single-threaded file-scanning. Thus, not a bug in USDX in my opinion. Just cache the files locally or copy them to a local folder. @jschwartzenberg

[s09bQ5]

The code to scan the songs on startup is not optimal for the following reasons:

1. We check all directories twice: one time for .txt songs and one time for .xml songs
2. We load songs after all directories have been checked, which might be bad for caching
3. For .txt files each file is loaded twice: one time to read the header and one time to read the notes
4. On *nix the Free Pascal runtime library ignores the d_type field returned by readdir and calls stat on all files in a directory to collect information we don't need.
5. On *nix the Free Pascal runtime library calls flock every time a file is opened.
6. There are lots of calls to access and stat that might not be cached on network filesystems.

I have a local branch where 1 and 3 are fixed. 5 can be worked around for NFS on Linux by mounting with the nolock option.

[jschwartzenberg]

@s09bQ5 would you mind pushing your branch? I'm very interested in trying it out. I will also try the nolock option.

[jschwartzenberg]

I just tried nolock and this pretty much fixed my issue! Start-up time has improved to a point where it's not hindering anymore on my ARM device. Thank you very much for that tip @s09bQ5!!

[cRaZy-bisCuiT]

@s09bQ5 Would you mind to merge your branch into master? Loading 8k songs is painfully slow here on Linux. It's like 5-8 minutes. Not sure if this is caused by some errors at start (not finding some videos and cover art).

[sw-tracker]

Has this been merged? How can I try this version, my UltraStar takes forever to load!

[carsten-engelke]

Has this been merged? How can I try this version, my UltraStar takes forever to load!

Hi, I was struggling too. Have you tried using Ultra Star Play? Worked fine for me so far. However I did not test it excessively.

[brodmo]

Has there been any progress on this?

[cRaZy-bisCuiT]

For me it's starting much quicker than in the past. This could as well being related to much faster systems including NVME SSDs nowadays.

[s09bQ5]

It's probably faster because you removed support for XML songs.

I still have that branch on another computer. Maybe I find some time this weekend.

[s09bQ5]

Made a pull request, but I don't think it will have a noticeable impact on the startup time: #819