orblivion / desert-atlas

Other
7 stars 1 forks source link

Compress search database to make grains much smaller #58

Open orblivion opened 1 year ago

orblivion commented 1 year ago

I realized that even with the base data (major cities, before any downloaded areas) it's already 183 megs. But if you gzip search.db it's 16 megs. No wonder, it's a bunch of text.

I bet there's an sql feature or plugin we could use to make it smaller. Especially since it's read-only 99% of the time. The goal is to find and implement such a thing for the search database.

Though if there's a notable performance hit for reading, it would give me pause. In that case maybe we could look into something awful like decompressing on startup.

Or maybe we just need to run VACUUM or set auto_vacuum. I'd guess auto_vacuum only happens on writes. I just tried it and it at least cuts size in half. https://www.sqlite.org/lang_vacuum.html