No results with short queries (although they do score in Tester in ES)

[bigrig2212]

Expected Behavior

Type "modify" in the Client chat UI. Should fulfill answer "Admin.001"

Using the default content in the admin tester, a search for "modify" returns answer "Admin.001" with a score of 1.97.

Actual Behavior

When you type "modify" into the Client chat UI, it says "Sorry, can you please repeat that?"

Steps to Reproduce the Problem

1. Just search for "modify"

Note: the problem goes away if you were to add "modify" as an alternate question for Admin.001.

I don't see the unsuccessful Client-side queries in the logs... am thinking they cut out somewhere in Lambda before getting logged? Or there is an ES score threshold? Would love a point in the right direction to debug. Thx.

[bigrig2212]

I think i've traced down the issue (but don't have solution). Lex is not recognizing these shorter questions as utterances that match the intent, so they're not being forwarded on to Lambda/ES (which is why they don't show up in the logs).

Is there a way to tell Lex to recognize everything?

It's more limiting that my example above conveys, which is quite terse. It fails more broadly. For example, the question "check status of reservation" fails also.

[rstrahan]

Hi @bigrig2212

Lex is not recognizing these shorter questions as utterances that match the intent, so they're not being forwarded on to Lambda/ES (which is why they don't show up in the logs).

Yes, you are right. QnABot provides a bunch of sample values for the slot type used in the Lex intent, which is sufficient for Lex to match many utterances in the form of questions, but it's not going to catch everything.

Take a look at the blog section on "Tuning the Bot’s Natural Language Processing" to see if that helps. The trick is to add your utterance to the list of questions, and then rebuild the Lex bot from the Content Designer as described in the blog. That should result in the new questions (or in your case, the word 'modify') being added to the slot type sample values list, which should then allow Lex to match the utterance to the intent, and pass it through to the QnABot Lambda handler.

Let me know if this addresses the issue.

Cheers Bob

[bigrig2212]

Hi. Yea, that solves the problem in an immediate sense. But limits, I think, the utility of ES under the hood. I'd think that if there's a high scoring response in ES, we'd want to show it. It makes the proposed ingestion utility much more functional: https://github.com/awslabs/aws-ai-qna-bot/issues/54.

[JohnCalhoun]

you are correct, the problem is that if Lex does not recognize the utterance then the lambda does not even get called and a static response is returned. In many lex use cases this makes sense but not for our use case.

also, since this a behavior of lex there are limited things that QnABot can do. :(

[bigrig2212]

Figured out a work around. If you add "how do you" in front of a query that is going to be (or is) rejected by Lex, it works. eg: "modify" - fails "how do you " + "modify" - gives the expected response.

So, could probably ping lex with the query and if there's a lex-error and the query is less than 3 words or so, could try again by prepending "how do you ". I think this works even if "how do you" doesn't make sense in the sentence. ES seems to sort it out just fine (although my test dataset is very small at this point).

[tiborATandoverlab]

First of all, brilliant work guys! Just out of curiosity, would it not give better results (through NLU) if those examples were intent utterances, as in a normal use case? Why are these examples slot type examples, any benefits over the utterances? And a cheeky question, if the ES search is the only intent (fulfilment), why bother using Lex at all?