How difficult would it be to modify lunr.js to lazy load a partial index?
For a large index covering hundreds of thousands of large documents, I've found it's impractical, and usually unnecessary, to load the entire index on the client side.
Inspecting the code, it looks like the two main data-structures in the index are fieldVectors and invertedIndex, which are both keyed by search term and document ID respectively.
Instead of requiring those entire data structures to be loaded, would it be possible to shard them, based on term, and then load the shards as needed during a query? This could drastically reduce index loading time as well as help lunr scale over larger collections of documents.
How difficult would it be to modify lunr.js to lazy load a partial index?
For a large index covering hundreds of thousands of large documents, I've found it's impractical, and usually unnecessary, to load the entire index on the client side.
Inspecting the code, it looks like the two main data-structures in the index are fieldVectors and invertedIndex, which are both keyed by search term and document ID respectively.
Instead of requiring those entire data structures to be loaded, would it be possible to shard them, based on term, and then load the shards as needed during a query? This could drastically reduce index loading time as well as help lunr scale over larger collections of documents.