search

olivernn / lunr.js

A bit like Solr, but much smaller and not as bright
http://lunrjs.com
MIT License
8.96k stars 548 forks source link

adding per field tokenizer #46

Open saliksyed opened 11 years ago

saliksyed commented 11 years ago

Added a per field tokenizer and changed the lunr variable to be explicitly set for window. (This prevents errors when running in strict mode)

olivernn commented 11 years ago

What is the use case for having individual tokenizers per field?

How does this affect searching too? Search queries are passed through the main lunr.tokenizer rather than the individual field tokenizers, would this not cause problems if the per field tokenizers are different? It'd be good to see some tests covering this specifically as well as how the per field tokenizers work.

I like the change for explicitly assigning lunr to window if it prevents warnings in strict mode, as I mentioned in the line comment it would be good to do a check so lunr can still work in environments where the global object is not window (node.js). I do think this should be a separate change though so if you could split this into two pull requests that'd be great.

saliksyed commented 11 years ago

Okay -- I can split into two pull requests and I see why you would want to check for window before assigning. The use case is that we have some fields that have strange formats (tokens seperated by "." notation) and want to use custom logic to split the strings before then passing results to default lunr tokenizer.

On Tue, Aug 20, 2013 at 9:27 AM, Oliver Nightingale < notifications@github.com> wrote:

What is the use case for having individual tokenizers per field?

How does this affect searching too? Search queries are passed through the main lunr.tokenizer rather than the individual field tokenizers, would this not cause problems if the per field tokenizers are different? It'd be good to see some tests covering this specifically as well as how the per field tokenizers work.

I like the change for explicitly assigning lunr to window if it prevents warnings in strict mode, as I mentioned in the line comment it would be good to do a check so lunr can still work in environments where the global object is not window (node.js). I do think this should be a separate change though so if you could split this into two pull requests that'd be great.

— Reply to this email directly or view it on GitHubhttps://github.com/olivernn/lunr.js/pull/46#issuecomment-22957841 .