olivernn / lunr.js

A bit like Solr, but much smaller and not as bright
http://lunrjs.com
MIT License
8.89k stars 545 forks source link

Enhancement: All prefix tokenizer. #356

Closed prasad83 closed 6 years ago

prasad83 commented 6 years ago

Powerup auto-completion or suggest with all-prefix tokenization.

Example:

Usage:

var idx = lunr(function(){
    this.ref('refid');
    this.field('field1');
    // ...

    // 1. Remove stemmer from pipeline.
    // 2. Activate tokenzier.
    this.pipeline.remove(lunr.stemmer);
    this.pipeline.remove(lunr.stopWordFilter);
    this.tokenizer = lunr.allprefixtokenizer;

    // ...add doc...
});
olivernn commented 6 years ago

The way that tokens are stored in lunr.TokenSet should allow good performance for prefix searches already. Do you have an example where this approach works better than just using a trailing wildcard suffix?

You could certainly implement this kind of approach as a pipeline function without having to replace the tokeniser.

Because of that I don't think this change should be part of lunr, though I encourage you to package it as a separate module/plugin.