microsoft / NLU.DevOps

Tools supporting continuous integration and deployment for NLU services.
MIT License
36 stars 24 forks source link

Determine how best to handle entity confusion results given speech mis-recognitions. #241

Open rozele opened 4 years ago

rozele commented 4 years ago

If I have a test case like:

{
  "text": "contact info for carole",
  "intent": "ContactInfo",
  "entities": [
    {
      "matchText": "carole",
      "entityType": "builtin.personName"
    }
  ]
}

And through speech transcription, we instead get something like:

{
  "text": "contact info for carol",
  "intent": "ContactInfo",
  "entities": [
    {
      "matchText": "carol",
      "entityType": "builtin.personName"
    }
  ]
}

Currently, this is reported as a missing entity.

rozele commented 4 years ago

This issue is probably not needed. We can't effectively test ASR -> entity results unless we are testing an end-to-end system and the entities are fully disambiguated.

The correct approach here is to label the ASR output instead of the expected output.

I will file another issue to consider whether we want to remove speech -> LUIS result testing.