be able to train dictionaries automatically

cldellow / datasette-scraper

Add website scraping abilities to Datasette

Apache License 2.0

60 stars 1 forks source link

Closed cldellow closed 1 year ago

cldellow commented 1 year ago

maybe-bad-idea: the seeder process could also train dictionaries.

on start up, get a list of all hosts in dss_fetch_cache
for each host, check if it has a dictionary
if it doesn't, and it has at least 100 entries in fetch_cache, train a dictionary for it
once the dictionary is trained, enqueue its existing hashes to be rewritten (see #43)

...maybe the seeder process needs to be rejigged? atm it's based on message passing, but perhaps it should just busy wait on database tables

cldellow commented 1 year ago

Huh, sqlite supports a callback on row change: https://www.sqlite.org/c3ref/update_hook.html

Doesn't look like sqlite3 supports it though. :(