liblcf is reading the whole RPG_RT.ldb twice on init

[fdelapena]

Some games have large database files and it can be slow parsing it completely.

Furthermore, the file is parsed twice: first to get some strings to detect the encoding with ICU, and then parsed again with the detected encoding.

For a proof of concept, try the Hero's Realm HTML5 version: https://easy-rpg.org/play/master/?game=heros_realm

It should allow to do selective parsing of data instead of eager loading.

[Ghabry]

Could be combined with a reader_struct refactor ;)

[fmatthew5876]

Lazy reading incurs a branch on every read because you always have to check if it's loaded or not. Most things in the database are used frequently enough you'd want them in main memory all the time.

One way we could speed up load times is by tricking the user. When starting up Player, we could load only the bare minimum required the operate the title screen and load menu. Then, while the user is on the title screen, we continue loading the rest of the database (and maybe the maps of his saves in order of most recent save) in a background thread. By the time the user picks new game or loads, a lot of, if not all of the lcf reading is done.

[fdelapena]

Since 0.6.2, games load even faster than RPG_RT. Maybe this issue can be closed.

[fmatthew5876]

Reading everything twice is still pretty bad and fixing this could speed up significantly more. For large games like Heroes Realm and HH3, the in memory LDB is the biggest memory hog of the entire engine.

Maybe it's possible to do something which only loads enough partial string data to figure out encoding and exits.

I'd like to leave this open until address the double load issue.

[Ghabry]

this will be solved when Saves use DBString, before we can't get rid of the Encode functions in the parsing code.