search

sciencefair-land / sciencefair

The futuristic, fabulous and free desktop app for working with scientific literature :microscope: :book:
https://sciencefair-app.com
MIT License
604 stars 52 forks source link

Distributed search index #66

Open mafintosh opened 7 years ago

mafintosh commented 7 years ago

Just an issue to track ideas on how to distribute the search index generated from the initialization step

blahah commented 7 years ago

In the simplest case we could just use hyperdrive as the kvstore instead of leveldb - but the latency involved in the lookups for the inverted index would probably be a huge bottleneck.

Worth trying as a POC I think, then we can optimise from there.

aschrijver commented 7 years ago

Some references:

blahah commented 6 years ago

I am in the process of implementing this on top of hyperdb. A lot of the hard problems are already solved by @mafintosh in that module. There are some tricky things to do with implementing an efficient full text search index on top it it - I hope to have them all ironed out with an initial release in the next few days.