Chatbot should be robust in handling of typos

[eleow]

Typos in questions from the user will limit the ability of the chatbot to respond with answers that are actually helpful.

One possible way we can tackle this is to do the following for each user input

• Tokenize using spaCy and send each word to rule-based spelling correction library such as Hunspell or DeepPavLov spelling correction or SymSpell, etc or even ML/DL models. This will result in a list of suggested replacements
• Compute similarity score of replacements with the original word
• Compute rank score for each suggestion based on certain criteria (eg frequency in English language based on n-gram frequency analysis of Google's Trillion Word Corpus)
• Combine and normalise to provide final score for each replacement, and select top n for each misspelled word
• Create list of permutations of different replacements for the entire sentence
• Finally send each of these to our backend for processing, and return the result with the highest confidence

(see article for more details)

Of course, the common alternative is to simply have a list of synonyms (also containing misspelled words) for each word that we think is important to us. This feature is supported by both DialogFlow and rasa out-of-the-box, but will not scale up.

[Shashankwer]

Just a thought. Instead of searching through all the dictionary items which might span across the different domain we can limit to words in our domain search. Also issues with incorrect words will mostly arise in intent classification/detection so based on those keywords search can be limited

[eleow]

Used synonyms generated by @Shashankwer , the system was more robust to typos. Details in the Project Report