Compounds and phrases in suggester

[twagoo]

Suggested by Dieter: it would be nice if the auto complete of the search box in the VLO user interface could suggest terms like 'middle ages', 'Noam Chomsky', 'eye tracking', ... Could the importer be adapted, e.g. by using a dictionary or searching for common n-grams, to include such phrases in the _suggester facet?

Maybe an easy alternative solution is to include all (common) values from a selection of facets (language, subject, genre, subject, ...) to the _suggester facet, or (maybe a cleaner and leaner approach) include those in the query for search terms from the client.

---

Transferred from Trac, issue #587

[twagoo]

(maybe a cleaner and leaner approach) include those in the query for search terms from the client.

To clarify, I believe that my thinking here was to extend the autocomplete service to combine the response of the Solr suggester (/suggest) with an additional query that gets matching values for selected facets. This probably was more of a brainwave than a fully fledged implementation proposal but it may be an alternative to consider if this cannot easily be fixed on the Solr server side.

[teckart]

After investing many hours in testing various Solr configurations (tokenizer/filter/spellchecker) I couldn't identify one that provides a satisfying result based on Solr directly (although a more experienced Solr user may come up with one). Main problems:

• ShingleFilter has the tendency to overgenerate terms. We have used this approach in the VLO before and IIRC it was disabled at some point for that reason. For the current set of files it also ran into a "GC overhead limit exceeded" exception.
• It is not 100% clear if using the content of specific fields (organisation, subject) is really helping for the intended behaviour. Many terms like "Noam Chomsky" are part of larger text content most of the time and therefore won't be contained in a simple field query. Furthermore, even facets like "subject" contain a lot of "ugly" content that we probably don't want to show to the user immediately (like "arabic language--grammar--early works to 1800" being the most frequent "subject" content for the prefix "ara".)
• We could use a manually created dictionary (wherever that may come from) by using a FileBasedSpellChecker and could combine it with some standard WhiteSpaceTokenizer-Index. However I couldn't find a configuration that actually worked as intended (still unclear what the problem is). On the other hand, if we had such a dictionary, we could inject it in the standard suggester output in AutoCompleteServiceImpl directly.

[twagoo]

Would things improve (in any way) if we switch from using the SpellCheckComponent (see solrconfig.xml to Suggester for this purpose? This is suggested in the official documentation (also see this comment) but I have not looked into the differences between the two to know what the required effort and consequences would be.

[twagoo]

We have looked into this and searched for expertise but were not successful in finding a usable solution. For now we are not going to put any substantial effort into this.

[menzowindhouwer]

Maybe interesting: "semantic query suggestions" http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1006058

[twagoo]

Not directly related but also see #193