bglw
commented
2 years ago Hello!
Boolean search operations isn't something I've touched yet, but I'm certainly happy for it to be in scope for the project. I haven't tidied things up for contributions yet, but if you're happy to delve in deep I can point you in some hopefully-good directions 😁
I think cats -"cute dogs" might be a better syntax to match Google operations, but I'm happy to inherit the operations of any sufficiently large incumbent search platform. Also, they don't have to be implemented now, but it would be good to keep operations like cats OR dogs in mind when building this.
An ad-hoc contributing guide
Pagefind sprawls a few Rust crates at the moment: pagefind, pagefind_stem, pagefind_ui, and pagefind_web. The main changes you'll need to make are to pagefind_web, which is the search webassembly module. Inside the pagefind crate lives the JS search API, which will also need some tweaks.
I'll give a super quick™️ rundown of the things I think you'll need to touch to implement this:
- pagefind/src/output/stubs/search.js (which is super tucked away 🥴) houses the JS search API. Mostly, this exists as a communication wrapper to the WASM.
It does also handle some parsing of the search term, so this could be a place to handle parsing the boolean operations. It depends on whether you want to implement that in JS or in Rust — both are fine choices.
- The search function is the entry point to look at: https://github.com/CloudCannon/pagefind/blob/5bde67679b0b5720c0274513210b9d21ed2f3c40/pagefind/src/output/stubs/search.js#L296-L378
- Currently it does a naïve check for a quote boundary on the search term, which it uses to run a slightly different exact search mode — we'll need to adapt this to handle mixed searches.
- pagefind_web/src/lib.rs is the entry point for WASM functions. The first thing to solve would be the
request_indexesfunction: https://github.com/CloudCannon/pagefind/blob/5bde67679b0b5720c0274513210b9d21ed2f3c40/pagefind_web/src/lib.rs#L118-L149- This is called from the search function above. It takes a string, splits it into words, and returns a list of the chunks that need to be loaded before it can run that search. So for
cats -"cute dogs", it will ultimately need to return a list of the chunks containingcats,cute, anddogs. The input to this function will depend whether JS or Rust is handling the boolean parse step. - The search function would be the next thing to read through: https://github.com/CloudCannon/pagefind/blob/5bde67679b0b5720c0274513210b9d21ed2f3c40/pagefind_web/src/search.rs#L99-L189
- It's fairly rudimentary as far as IR goes — it builds some bitmaps, intersects them, and then weights the results. Again, the input to this function will depend whether JS or Rust is handling the boolean parse step.
- You'll also see an
exact_searchfunction above that function — we'll need to combine these operations for the boolean world where part of a search might be exact, but that shouldn't be a big challenge.
- This is called from the search function above. It takes a string, splits it into words, and returns a list of the chunks that need to be loaded before it can run that search. So for
I don't have time at the moment to cover everything, but if you have a skim of those files/functions and let me know any questions you have I can help out further. I can also cover the testing/local dev setup tomorrow (if you're game to give this a crack 🙂)
Let me know — hopefully that isn't too daunting an info dump!
c-w
Sailsman63
Hi there. Thanks for releasing this amazing project! 🙇 I have a question about more advanced search queries:
Is it possible to exclude certain terms from a search? E.g. I'd like
cats ~"cute dogs"to return all documents that contain the word "cats" but not the phrase "cute dogs".If this functionality isn't implement yet but you'd be happy to have it in the scope of the project, I'm happy to work on it and open a pull request if you could provide me with some guidance on how to best go about this.