search

plicit / scoop-search-multisource

Searches Scoop buckets: local, remote, zip, html
MIT License
39 stars 0 forks source link

use the sqlite db file for better performance #4

Open phanirithvij opened 1 year ago

phanirithvij commented 1 year ago

The https://rasa.github.io/scoop-directory/search site uses an SQLite database https://rasa.github.io/scoop-directory/scoop_directory.db, source code for generating it and the client SQL queries.

plicit commented 1 year ago

Thanks for the suggestion. I can download, cache, and search rasa's sqlite db just to allow an additional source type, but I doubt it would improve performance much beyond the current 500ms for searching rasa's html. e.g. is the following slow for you?

scoops -source :rasa ziglang

The biggest performance problem I've had is the long delay on first run when searching through all of the .json in the local buckets directories. Once Windows caches those files, then searches are fast.

Theoretically, I could keep an sqlite database summarizing all of the local json, but then I'd still have to scan the buckets on every run, compare timestamps, and keep it in sync.