search

quickwit-oss / tantivy

Tantivy is a full-text search engine library inspired by Apache Lucene and written in Rust
MIT License
11.33k stars 621 forks source link

Client side web search? #1413

Open chrisspen opened 1 year ago

chrisspen commented 1 year ago

Does Tantivy support client side web search, specifically through lazy loading or incrementally loading slices of an index?

I see in the docs it supports incremental index creation, but there's no mention of client side search. Is this only meant to run on the server side?

I have a large (~1 million) collection of documents that I want to make searchable via the browser with no search server, and it's impractical for the user to load a hundreds-of-MB search index.

ChillFish8 commented 1 year ago

Realistically, you'd probably have a hard time either way with the amount of data required, it'd simply be easier using something like a Lambda if you really didn't want a dedicated server running.

The requirement is basically, you need WASM support which isn't supported right out of the box: see https://github.com/quickwit-oss/tantivy/issues/541 and https://github.com/quickwit-oss/tantivy/pull/1067

Another example that might be of some interest to you https://jakejscott.com/full-text-search-for-dynamodb-using-lambda-efs-tantivy-and-rust

chrisspen commented 1 year ago

I was hoping to avoid any proprietary tech too, aside from large scale cheap storage like S3.

Since most client-side search uses static index files, I was hoping to find something that supports sharding the indexes based on keyword range, and then loading them dynamically based on the search terms.

I'm working on a proof of concept on my own, but I find it hard to believe I'm the only one who would find this useful. I've used DynamoDB and Lambda, and they're good for certain things, but they're still a lot more expensive than loading simple static index files and letting the client due the heavy lifting.