detect unsupported requests and handle difference from low confidence

[aeweidne]

Goals:

Reduce the number of low confidence matches. Have a bot that appears smarter to users. We should create an NLC classification with unsupported statements, let's call it unsupported.statements. This classification will match statements we know aren't supported, for example statements about sports, food, and other random categories.

Now, when any of these unsupported statements are matched, we don't have to report it as low confidence, instead, we have a high confidence that we don't care about that statement.

Instead of ignoring the request, or posting low confidence as we do now. We should reply to the user with something like I'm only trained in Bluemix Dev Ops.

[aeweidne]

@jpadilla

Starting to take at look at this; wouldn't it be easier to have a lower bound confidence threshold that would catch all these statements? I think having a list of unsupported statements is unrealistic. If the confidence is low enough, we can determine that it is unsupported. We use this logic in hubot-ibmcloud-service-suggest

[aeweidne]

We need to brainstorm and discus this issue more. I agree with @reicruz that having a list of unsupported statements is unrealistic and even dangerous because a new project can decide to implement those statements and it will be competing against what we defined as unsupported.

I want to understand better how the concept of ultra low confidence would work here. What do you think would be a good threshold? Remember that we are trying to capture low confidence results to improve the Bot's training, but it could be that at some point the confidence it so low that we could interpret it as "it's not even close to what we know."

Trying to understand it better, I asked my bot Do you know about sports? and the results were:

1. [59.02%] nlc.help
2. [10.67%] ibmcloud-weather.alerts
3. [ 6.87%] ibmcloud-weather
4. [ 4.42%] ibmcloud-weather.conditions
5. [ 3.18%] ibmcloud-weather.temperature