Closed aconradi closed 3 years ago
Yeah, it would be nice to allow custom tokenizers. It's already gotten a bit disorganized with e.g. tokenize_chinese
. It's probably about time to consider a 3.0 with a better API that doesn't necessarily mirror the JS one so much.
I would take a patch that adds a hook somewhere for a custom tokenizer in the mean time. Index::add_doc_with_tokenizer
seems fine given the limited flexibility of the Pipeline
currently.
Fixed by #37
I'd like to see some API to build the index with a custom tokenizer. For my case I index documentation of (among other things) commands for a command-line-interface. Those commands often have names like
foo-bar
, and users expect to be able to search for and find the documentation for such commands. With the default tokenizer the index instead contains the two tokensfoo
andbar
. I did not find any API to change the default tokenizer.I have a local patch to add an
Index::add_doc_with_tokenizer
method. I'd be happy to share if I can figure out the company policy for doing so. But I'm also not sure if that is the best way to add such functionality.