search

wit-ai / wit

Natural Language Interface for apps and devices
https://wit.ai/
941 stars 91 forks source link

Issue in identifing intent which has no relation with keyword #2323

Closed prajisha closed 2 years ago

prajisha commented 2 years ago

Hi

We are facing an issue with the wit workspaces. If we input a random text "aaa" or "bbb" or "test", we are getting an intent which has no relation with the keyword. Can you please look into the issue with high priority, as it is affecting our business. 'aaa' returning 'hi' intent. 'test' returning 'tentative' intent. We are facing this issue with all worspaces and here is the app id of one of the workspace. App ID : 4344353432292657

Thanks

image image image image

nishsinghal20 commented 2 years ago

@prajisha Yes sure, we'll take a look into this. Thank you for your patience!

patapizza commented 2 years ago

Hi @prajisha,

The confidence level for the provided inputs is very low. While we do provide the confidence and auto-populate the intent to speed up the annotation, we don't return them under 50% confidence in the API runtime. The 50% confidence threshold would optimize for recall over precision. For higher quality, I encourage you to use the model insights on the intent page to find the best confidence threshold for your use case using precision/recall, and ignore API predictions below that threshold. Most apps should probably use at least a 85% confidence threshold, but again that depends from one product to another one.

Hope this helps.

prajisha commented 2 years ago

Hi

When i tried random inputs via API am getting intents in the response even if the confidence level is too low.Below are two examples,

wit_response {"_text":"test","entities":{"intent":[{"confidence":0.22642825830761,"value":"tentative"}]},"WARNING":"DEPRECATED","msg_id":"03pVd4dOzh6iurirG"}

wit_response {"text": "xxxx","intents": [{"id": "381104606705616","name": "authenticate","confidence": 0.215}],"entities": {},"traits": {}}

Till earlier last week, we were not getting the intents in response if its confidence level is too low. The issue seems to occur from thursday.

Thanks

patapizza commented 2 years ago

@prajisha Oh indeed. Thanks for reporting. Let me look into this.

GuilhermeXlab commented 2 years ago

We have the same issue here..

People names has been identified with "curse" without any correlation.

Client names like "Guilherme" or "Marinalda" have 90% to 99% of confidence. Equally with randoms: https://api.wit.ai/message?q=adwefaefsdf

Our AppId: 4151339554959163

Thanks!

prajisha commented 2 years ago

@prajisha Oh indeed. Thanks for reporting. Let me look into this.

Hi @patapizza, Do you have any updates on this? This is badly affecting our customers.

prajisha commented 2 years ago

@prajisha Oh indeed. Thanks for reporting. Let me look into this.

Hi @patapizza, Any update on this?

doggeral commented 2 years ago

@prajisha Oh indeed. Thanks for reporting. Let me look into this.

Hi @patapizza, Any update on this?

We're working on the fix now. Will keep you updated.

prajisha commented 2 years ago

@prajisha Oh indeed. Thanks for reporting. Let me look into this.

Hi @patapizza, Any update on this?

We're working on the fix now. Will keep you updated.

Hi @patapizza Any update on this issue? This is badly affecting our customers for more than a week.

doggeral commented 2 years ago

@prajisha verified on wit.ai, your issue has been fixed!

anjaly commented 2 years ago

Hi @doggeral

We are currently facing this issue in the following AppId: 623134488936494. We are getting intents with high confidence for non trained inputs.

Can you please check.

GuilhermeXlab commented 2 years ago

Please, our app have the same issue: https://api.wit.ai/message?q=marinalda = (99,1% confidence) Our AppId: 4151339554959163

Thanks!

anjaly commented 2 years ago

Hi @doggeral

We are currently facing this issue in the following AppId: 623134488936494. We are getting intents with high confidence for non trained inputs.

Can you please check.

@doggeral @patapizza Any update on this.

doggeral commented 2 years ago

@GuilhermeXlab @anjaly You can add the the utterance as out of scope and retrain the app.

GuilhermeXlab commented 2 years ago

we have already done this for many cases, however, it is impossible to predict all possible words that may be used. every day many phrases are identified as out of scope and categorized, and this behavior is new to us.

All these examples have high confidence without any explanation:

https://api.wit.ai/message?q=ambulancia https://api.wit.ai/message?q=amendoim https://api.wit.ai/message?q=mauricio https://api.wit.ai/message?q=tulio

Our AppId: 4151339554959163

In a restart attempt, we cloned the app and it behaved the same for a while. Soon after the fix announced here, it stopped working sending us the return 500: { "error": "Something went wrong. We've been notified", "code": "unknown" }

new AppId: 661807068294184