Steming is giving too wide results

[morchickit]

I queried GrantNav for "Clinical" and got results with the word clinic. I DID NOT want to get the word clinic at all... how can we make sure it doesn't make it too wide?

[robredpath]

The options are, I think:

• Export the data to another tool and use that tool to refine searches in cases where stemming has produced unhelpful results
• Try out other stemmers (see https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-stemmer-tokenfilter.html). It looks like the light_english stemmer might do what we want. But, of course, this would require some testing to see if it had its own issues. The stemmer wasn't a conscious choice AFAIK, but it's something I'd be cautious about changing)
• Write our own custom stemmer (probably overkill).

Something we can consider as part of future GN work, for sure.

[mariongalley]

https://trello.com/c/tFLVz8eh/213-stemming-can-cause-confusing-or-inaccurate-results-in-grantnav

[mariongalley]

Issue description:

When users are trying to make a specific search, elasticsearch will do fuzzy matching to look for similar words e.g. searching for "community organiser" will also return results containing "community organisation". This happens even when the user uses quotes, and when they use the AND operator, such as "community AND organiser".

This is an issue as "community organiser" and "community organisation" are very different things.

Possible solutions:

• using quotes prevents fuzzy matching/synonyms being detected (I would have thought that the use of quotes would already return exact results, but apparently it does not)
• an operator which turns off fuzzy matching on the current search e.g. "EXACT community organiser"
• a toggle in the filters which allows you to choose between exact matching vs fuzzy matching

It's clear how the world will be different when the work is completed

Users will have more control over whether GrantNav behaves more like Google i.e. "give me everything that might be related to the topic, no matter how tenuous" or more like a specialist search engine i.e. "give me all the items that exactly match my criteria"

[mariongalley]

@michaelwood In my latest comment I described a few possible solutions, all of which essentially boil down to having the option to turn off stemming/fuzzy matching. Is it possible to do this in GrantNav?

[michaelwood]

I think we could have an option to turn off the fuzzy matching. This would involve us having an internal field which isn't analysed and could be switched to via the front end.

That said I'd like to see the quotes method working, as you said this mirrors other known search paradigms and is expected behaviour. This may have been a regression when we changed version of ElasticSearch as I am fairly sure this used to work we may need now need to set the parameter https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#fuzziness based on whether there are quotes in the search string.