search

Azure-Samples / cognitive-services-language-understanding

Samples for the Language Understanding Intelligent Service (LUIS)
MIT License
85 stars 139 forks source link

Is bing spell check for integration with luis helpful with the dispatch pattern and is it useful or harmful to entity assurance? #23

Open xtianus79 opened 4 years ago

xtianus79 commented 4 years ago

An article here gave some caution to using the Bing spellchecker api for washing all of your user utterances into your bot channel.

Link to article here

Point being the entity could become unravled when the spellchecker is turned on? So if you have an obscure word style entity starblazingIT this might be reconstructed to star blazing IT. Thus harming the entity.

With that said the article was written in 2016 so perhaps this concern has been addressed but I can't find any documentation on it.

The documentation I did find on it gave an outlet for ignoring spelling mistakes and simply label utterances and the LUIS would learn proper spelling as well as typos... That is a loaded statement. What does that mean exactly and is luis really learning proper spelling or just adding the bad spellings in there as an alternative answer?

Because if that is the case then one would have to watch misspelled utterances and add in variations all of the time. Or would Luis overtime training learn that a particular word can have variations and be misspelled and to add that in with the correct intent model of utterances?

Here is a link to the article and quote below:

Ignore spelling mistakes If you don't want to use the Bing Spell Check API v7 service, you can label utterances that have spelling mistakes so that LUIS can learn proper spelling as well as typos. This option requires more labeling effort than using a spell checker.

Overall what is the best practice of word misspelling correction for your luis chatbot application?

diberry commented 4 years ago

@aliciaep Can you answer this one?