ibm-cloud-solutions / hubot-ibmcloud-nlc

Adds a framework to enable Natural Language interactions in your Hubot scripts
http://www.ibm.com/
Other
11 stars 1 forks source link

detect unsupported requests and handle difference from low confidence #9

Open aeweidne opened 8 years ago

aeweidne commented 8 years ago

Goals:

Reduce the number of low confidence matches. Have a bot that appears smarter to users. We should create an NLC classification with unsupported statements, let's call it unsupported.statements. This classification will match statements we know aren't supported, for example statements about sports, food, and other random categories.

Now, when any of these unsupported statements are matched, we don't have to report it as low confidence, instead, we have a high confidence that we don't care about that statement.

Instead of ignoring the request, or posting low confidence as we do now. We should reply to the user with something like I'm only trained in Bluemix Dev Ops.

aeweidne commented 8 years ago

@jpadilla

Starting to take at look at this; wouldn't it be easier to have a lower bound confidence threshold that would catch all these statements? I think having a list of unsupported statements is unrealistic. If the confidence is low enough, we can determine that it is unsupported. We use this logic in hubot-ibmcloud-service-suggest

aeweidne commented 8 years ago

We need to brainstorm and discus this issue more. I agree with @reicruz that having a list of unsupported statements is unrealistic and even dangerous because a new project can decide to implement those statements and it will be competing against what we defined as unsupported.

I want to understand better how the concept of ultra low confidence would work here. What do you think would be a good threshold? Remember that we are trying to capture low confidence results to improve the Bot's training, but it could be that at some point the confidence it so low that we could interpret it as "it's not even close to what we know."

Trying to understand it better, I asked my bot Do you know about sports? and the results were:

  1. [59.02%] nlc.help
  2. [10.67%] ibmcloud-weather.alerts
  3. [ 6.87%] ibmcloud-weather
  4. [ 4.42%] ibmcloud-weather.conditions
  5. [ 3.18%] ibmcloud-weather.temperature