Urgent Bug - Wit.ai model not returning empty intents for utterances trained as ‘Out of Scope’

[s994n]

Do you want to request a feature, report a bug, or ask a question about wit? bug

What is the current behavior? The Wit.ai model is not correctly returning empty intents for utterances that have been trained as ‘Out of Scope’.

This behavior seems to be similar to previously reported issues, https://github.com/wit-ai/wit/issues/2007 and https://github.com/wit-ai/wit/issues/2027, which were closed owing to inactivity.

Description for my app and a minimal demo: My app: After extensive training with over 900 utterances (fake email addresses) marked as 'Out of Scope', there is no change in the intent associations made by the Wit.ai model. Specifically, approximately 50-60% of these utterances continue to be associated with the 'tryAgentMode' intent, exactly as they were before training. This indicates that the training process is not impacting the model's behavior. This unchanged behavior persists even after removing and re-adding the wit/email entity, varying the training data, and utilizing both the dashboard and API for training.

Also to clarify: The utterances are marked as 'Out of Scope' on the Wit app’s 'Utterances' page, but those same utterances are associated with a ‘tryAgentMode’ intent in responses from Wit, via both the API’s /message endpoint and when interacting with the ‘Utterance’ input on the ‘Understanding’ page of the Wit app.

Minimal demo: The same result as described above is observed in a minimal demo, where only a few utterances are trained as Out of Scope. After training, Wit still returns an intent of 'tryAgentMode' for the utterances that have been trained as 'Out of Scope'.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.

My app:

• Injected approximately 1000 fake email-address utterances into test apps, marked as 'Out of Scope'.
• Attempted to train the model to not associate these utterances with any specific intent (trained as 'Out of Scope').
  • Checked that the utterances are marked as 'Out of Scope' on the Wit app’s 'Utterances' page
• Observed that the model continued to associate these email utterances with the 'tryAgentMode' intent.
• Tried removing and re-adding the wit/email entity, and varying the training data quantity.
• The issue persisted across all these variations.

Minimal demo:

• Trained a few utterances to have the intent ‘tryAgentMode’.
• Identified four additional utterances currently assigned an intent of ‘tryAgentMode’.
• Trained these four additional utterances as Out of Scope.
  • Checked that the utterances are marked as 'Out of Scope' on the Wit app’s 'Utterances' page
• After the training, Wit still returns an intent of 'tryAgentMode' for the utterances trained as Out of Scope.

What is the expected behavior? As-per Wit.ai's documentation:

Out-of-scope utterances for an intent or for a trait are utterances that do not have the intent or trait.

Therefore, the expected behavior for the Wit.ai model, after training with utterances marked as 'Out of Scope', is to correctly return empty intents for these utterances. This means that when an utterance is designated as 'Out of Scope' during training, the model should not associate it with any predefined intents, such as 'tryAgentMode' or others. Instead, the response for these utterances, when queried via the API or tested in the Wit.ai dashboard, should reflect their 'Out of Scope' training by returning an empty intent. This behavior would indicate that the model accurately recognizes and categorizes these utterances as outside the scope of its existing intent definitions.

If applicable, what is the App ID where you are experiencing this issue? If you do not provide this, we cannot help. My app: 681776129989206 Minimal demo: 1519823255483710 3523853844520900

[patapizza]

Thanks for reporting. It seems like the out of scope detection doesn't work well for utterances consisting of only email addresses. Indeed, it's not really an intent but an entity, and a given entity might suit different intents based on the conversation context. I'm not able to access the minimal repro app - have you deleted it?

[s994n]

Hello @patapizza, thank you for looking at this.

Apologies for the confusion: Yes, the ID I provided was incorrect; it belonged to an old app that I had deleted 🤦‍♂️.

The correct ID is 3523853844520900, and I've updated it in the previous message. Hopefully you can now access this, but please let me know if there are any further issues.

Also, I appreciate your clarification regarding the matter being more related to an entity rather than an intent. That makes sense. Thank you.

[s994n]

Hi @patapizza, I'm wondering whether you're able to provide a rough estimate of how long the issue might take to investigate from your end? My decision to implement a workaround in my codebase depends on this timeframe. If it's a short wait, I'll hold off; if not, I'll proceed with the workaround. Any rough estimate would be greatly appreciated.

[s994n]

Hi @patapizza, just keeping this on your radar. I'd be really grateful for any thoughts you have on my question above.

[s994n]

Hi @patapizza, any update on this?

[hvo408-brightplan]

I am experiencing the same issue as well. I trained some utterances as out of scope several time already, but it keeps returning incorrect intents for them. I am expecting empty intents for these utterances.