search

wit-ai / wit

Natural Language Interface for apps and devices
https://wit.ai/
936 stars 91 forks source link

wit returns intents with low score or wrong intents #2601

Open MeranTumeh opened 1 year ago

MeranTumeh commented 1 year ago

Do you want to request a feature, report a bug, or ask a question about wit? bug

What is the current behavior? I have 2 apps, sometimes the expressions return wrong intents or the correct intent with low score, this happens randomly and to different expressions

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem. can you retrain the 2 apps from your side so this doesn't happen again?

What is the expected behavior? it should always return the correct intent and with high score

If applicable, what is the App ID where you are experiencing this issue? If you do not provide this, we cannot help. 298027741480469 656232561652881

ChrisyShine commented 1 year ago

Wit.ai is predicting the intents with a model. The model quality will be affected by the training data and the amount of training data. We cannot ensure that the annotated data will be predicted exactly the same as the annotation. It is limited by the nature of the model. Can you provide some detailed examples to show the problem? E.g. what is the utterance, and is the utterance annotated? What is the expected intent and what is the current prediction result? Most of the time, adding more training data or review the existing training data to reduce ambiguity will help to improve the prediction accuracy.

MeranTumeh commented 1 year ago

Hi @ChrisyShine here is an example : The utterance "السلام" should return this intent "AssalmAleikum" , what happens that sometimes it returns it correctly and sometimes not

ChrisyShine commented 1 year ago

The prediction for "السلام" seems very stable and the intent confidence is more than 0.9. How did you judge the result is wrong? Are you using the HTTP /message API?

MeranTumeh commented 1 year ago

@ChrisyShine yes correct, its confidence is high but what happens with us that sometimes and in a random way the confidence becomes low and we get a wrong intent this happened with some expressions not just "السلام" and again, it happens randomly

that is why we thought that maybe you can retrain the app from your side so we don't face this issue again and yes I do use the HTTP/message API

MokhtarKhorshid commented 1 year ago

When it got it wrong, what intent did you get back?

On Sun, Mar 19, 2023 at 12:23 AM MeranTumeh @.***> wrote:

@ChrisyShine https://github.com/ChrisyShine yes correct, its confidence is high but what happens with us that sometimes and in a random way the confidence becomes low and we get a wrong intent this happened with some expressions not just "السلام" and again, it happens randomly

that is why we thought that maybe you can retrain the app from your side so we don't face this issue again and yes I do use the HTTP/message API

— Reply to this email directly, view it on GitHub https://github.com/wit-ai/wit/issues/2601#issuecomment-1475135256, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABTCDDMENTI6QNHDSNBB5O3W42X6BANCNFSM6AAAAAAV2GPFKY . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Mokhtar M. Khorshid

MeranTumeh commented 1 year ago

@MokhtarKhorshid humantransfer

ChrisyShine commented 1 year ago

@MeranTumeh Can we focus on "السلام" first? Can you copy/paste the full request and response (mask the access_token) for the case that this query returns a wrong intent? And offer a time period when this error case was generated. I want to learn how the full request/response looks like to get more clues on the issue. The problem is weird. It is possible that the endpoint returns 500 errors in case the server doesn't work well. But it is hard to tell why the server returns wrong intent randomly. The full request/response may be very helpful.

MeranTumeh commented 1 year ago

Hi @ChrisyShine, so far I didn't face the issue with "السلام" but I faced it today with "good morning" in this app id "298027741480469" Here are the request and the response as you can see in the screenshot, it returns the correct intent "GoodMorningEN" but the score is low : image

This happened today 22-3-2023 at nearly 11:30 am

also at 11:16 am today the score for it was "0.24851084" while before some days on 16-3-2023 at 9:37 the score was high "0.73924893"

ChrisyShine commented 1 year ago

@MeranTumeh I noticed that, in your training data, "Good Morning" is annotated as OutOfScope. This will impact the prediction of "good morning". I think the confidence difference is caused by the data difference. Every time, a new utterance is annotated, the model will be retrained. Can you double check your training data and make the data more consistent?