Search - Word Autocomplete

[kand617]

As a user, when I type my partial words in to search box, I would like the system to provide suggestions. This will free me from remembering exact names..... Now given that I could use search operators such as valu*, the system suggesting the word would be a nicer user experience.

Note this is not a prediction system like google search. But rather a simple word autocomplete. For example: startD would give startDate, startDay, creditStartDate etc

[kand617]

Possible Naive Implementation:

1) Process each document, to form global dictionary of words and store that in a table. 2) Each word becomes an row in hibernate search 3) As the user types a word, we select the partial word and return the top 5 matches complete words

The downside of this, there are a lot of rows, almost an overkill maybe. Alternatively.... we could do the following.

1) Create a global list of words, and maybe store on client side... 2) Use a javascript library to perform the search. Fuzzy search on a list of strings. e.g https://github.com/mattyork/fuzzy or http://fusejs.io/

Downside:

• Need to transfer all the global list of words.

Plus side:

• No data strain on the server, not having 10K records for 10K words.
• Reduced calls to the server

[sudhirtumati]

Storing the dictionary in database and querying it every time would be a overkill, I agree. At the same time, letting client manage it presents different set of issues

• When is the dictionary pushed to client? Once per session?
• How is it refreshed with any changes made on server? Should user reload page to get latest dictionary?

Does hibernate search offer any features to address this use-case? In my view, this should be explored first before any alternate solutions.

[kand617]

I could be wrong, but Hibernate Search functions work based off an entity and a search on attribute(s) can only return entities as a whole. This is useful if the aim of auto complete is based on the title (not our case).

With the client side approach,

• This dictionary can be stored in local storage, can be queried using if-modified-since approach
• The check for updates can be done if the last check was done n minutes ago. Maybe 15 minutes or more.

[kand617]

Actually I have another solution which might be a lot cleaner.

• Store the entire list on server side (persisted to a file or db(single row, single column), and maintained in memory on start up)
• Use a fuzzy search library on server side
• API returns a list of strings, say max 5.

Pros:

• Super simple, no complex api
• Don't need to worry about local storage size limits, compression, cache updates
• Easy to test

Cons

• More API traffic, but that should be ok
• Slightly larger heap on server side

This library looks ideal: https://github.com/xdrop/fuzzywuzzy

FuzzySearch.extractTop("goolge", ["google", "bing", "facebook", "linkedin", "twitter", "googleplus", "bingnews", "plexoogl"], 3)
[(string: google, score: 83, index: 0), (string: googleplus, score: 63, index:5), (string: plexoogl, score: 43, index: 7)]

[kand617]

@sudhirtumati you mentioned you know of a way to play around with Lucene index to. Care to suggest the alternative?

[sudhirtumati]

@kand617 Here you go. These might get you started

• https://dzone.com/articles/hibernate-search-based
• https://stackoverflow.com/questions/5361941/autocomplete-using-hibernate-search

[sudhirtumati]

https://wiki.apache.org/lucene-java/TheBasics

[kand617]

Issue resolved with PR #37 Enhancements to the quality will be done by subsequent PRs