search

btrask / stronglink

A searchable, syncable, content-addressable notetaking system
Other
1.04k stars 39 forks source link

Intersection query caching #22

Closed btrask closed 9 years ago

btrask commented 9 years ago

Due to the strict requirements of our queries, intersections are necessarily slow in cases where an extremely common clause is joined with an extremely rare one. I don't see any way to speed this up aside from caching.

We should be able to do caching intelligently, basically incrementally caching of only the items needed to satisfy a given request. Caches never have to be invalidated and can just be slowly built forever. So that's nice.

Where to store these caches is an open question. Storing them in the main database seems like a bad idea for several reasons. Maybe a second database or separate files. Durability is not necessary.

Note that LevelDB eats file descriptors for lunch (we currently cap it to 1/3 of the process's limit). But that's not a problem specific to caching.

btrask commented 9 years ago

This idea is probably superseded by #44.