search

GuidanceOfGrace / CLASSIC-Fallout4

Crash Log Auto-Scanner & Setup Integrity Checker for Fallout 4 / Buffout 4. Tool that extensively scans Fallout 4 game & mod files and Buffout 4 crash logs, then provides troubleshooting advice depending on what it finds.
8 stars 4 forks source link

Make FormID Lookups More Scalable #75

Closed evildarkarchon closed 10 months ago

evildarkarchon commented 1 year ago

Using an SQLite database to make FormID Lookups much faster, with my ~250MB database, I went from 60 seconds with just 29 logs down to about 17 seconds. The standard DB should be even faster. You will have to generate the DB yourself because the database is too large for GitHub without using Git LFS. There is a script to generate the DB in the new CLASSIC Tools directory. I also have it prepped and ready to go for Skyrim support. This PR includes the changes from #74.

evildarkarchon commented 1 year ago

After further inspection, I accidentally removed the header at some point, it wasn't missing before I started tinkering :(

evildarkarchon commented 1 year ago

Screenshot 2023-10-13 054217 Database size is 429MB (it would be about half that without the index). Without the index it would take ~25 seconds and searching the text files took more than a minute (with fewer logs). The good news is that the database files compress very well. Screenshot 2023-10-13 055240

evildarkarchon commented 1 year ago

Forgot to mention, the reason the DB in the screenshot is so big is that I've added my whole load order and a select few other mods in there as well. The source text file was a little bigger than the vanilla one.

evildarkarchon commented 1 year ago

Screenshot 2023-10-14 044249 Here's a version of the screenshot with just the vanilla form-ids.

evildarkarchon commented 1 year ago

Screenshot 2023-10-17 010828 Did another "benchmark" using my old crash log archive (over 500 logs), and here are the results with the SQLite DB and FID lookup turned on. This is about the amount of time it took for me to do a log scan before I did my regex thing back in the CLAS days.

evildarkarchon commented 1 year ago

The database(s) are now optional. If it doesn't find the game's formid database, it will use the old method of looking up formids. I also made it so that there will be a database per game since it's the easiest way to ensure that we aren't making queries against a non-existent table. Plus, there's less data to sift through to find what we're looking for.

evildarkarchon commented 10 months ago

Superseded by #76