Complex intent data

[daniele-athome]

I had an issue recently that forced me to modify HASS event data generation like this: daniele-athome/rhasspy@bd644f2236c7db02e596f8bd94ac868ad80562d5

The issue arised when I needed complex data provided by Duckling through Rasa. For example, when asking to start a timer, I would have a duration slot in either hours, minutes or seconds (or even mixed one, like "2 minutes and 30 seconds"). Duckling would correctly recognize this, but in the event data received in HASS, I didn't have the time units, just the numbers (2 and 30). Maybe my approach wasn't the best one, I'm just giving you an idea on what the problem is.

Possibly related to #40.

[synesthesiam]

The HASS event should contain _text and _raw_text properties with the complete spoken sentence. Do you have an example of your intent? Depending on how you wrote it, I could see Rhasspy either including or excluding the unit names from the slot values.

[daniele-athome]

The HASS event should contain _text and _raw_text properties with the complete spoken sentence.

It does, but I'd prefer that the event comes already completely processed by Rhasspy (I mean this is part of the Rasa NLU pipeline anyway).

Do you have an example of your intent?

This is my configuration in sentences.ini:

[SetTimer]
(imposta|avvia|metti) [un] timer [(per|di)] (2 minuti e 40 secondi){duration}
(imposta|avvia|metti) [un] timer [(per|di)] (1 minuto){duration}
(imposta|avvia|metti) [un] timer [(per|di)] (13 secondi){duration}

This is the raw intent data without my patch:

{
   "entities" : [
      {
         "confidence" : 0.9883932124,
         "end" : 29,
         "entity" : "duration",
         "extractor" : "CRFEntityExtractor",
         "start" : 18,
         "value" : "due secondi"
      },
      {
         "additional_info" : {
            "type" : "value",
            "value" : 1
         },
         "confidence" : 1,
         "end" : 8,
         "entity" : "number",
         "extractor" : "DucklingHTTPExtractor",
         "start" : 6,
         "text" : "un",
         "value" : 1
      },
      {
         "additional_info" : {
            "normalized" : {
               "unit" : "second",
               "value" : 2
            },
            "second" : 2,
            "type" : "value",
            "unit" : "second",
            "value" : 2
         },
         "confidence" : 1,
         "end" : 29,
         "entity" : "duration",
         "extractor" : "DucklingHTTPExtractor",
         "start" : 18,
         "text" : "due secondi",
         "value" : 2
      }
   ],
   "intent" : {
      "confidence" : 0.9984958172,
      "name" : "SetTimer"
   },
   "intent_ranking" : [
      {
         "confidence" : 0.9984958172,
         "name" : "SetTimer"
      },
      {
         "confidence" : 0.0005479114,
         "name" : "GreetingGeneral"
      },
      {
         "confidence" : 0.0002262423,
         "name" : "GetDate"
      },
      {
         "confidence" : 0.0001869724,
         "name" : "GreetingGoodMorning"
      },
      {
         "confidence" : 0.0001819576,
         "name" : "GetTime"
      },
      {
         "confidence" : 0.0001679873,
         "name" : "AllLightsOff"
      },
      {
         "confidence" : 0.0001459537,
         "name" : "GreetingGoodNight"
      },
      {
         "confidence" : 3.90133e-05,
         "name" : "SetDinnerEating"
      },
      {
         "confidence" : 7.7541e-06,
         "name" : "GetExternalTemperature"
      },
      {
         "confidence" : 3.854e-07,
         "name" : "LivingUtilityLightOn"
      }
   ],
   "raw_text" : "avvia un timer di due secondi",
   "siteId" : "default",
   "slots" : {
      "duration" : 2,
      "number" : 1
   },
   "speech_confidence" : 1,
   "text" : "avvia un timer di due secondi",
   "time_sec" : 0.0218460559844971,
   "wakeId" : ""
}

See the slots data? "duration" is just "2" but HA has no clue if those are seconds or minutes or hours. That information can be found however in an entity provided by DucklingHTTPExtractor.

[daniele-athome]

In the meantime I've modified Rhasspy once more to better pack those additional slot information, but I think a better proposal for this would be #40. If you agree, feel free to close this issue as duplicate.

[synesthesiam]

Oh, I see what you're saying now. I think this is something unique worth considering for the HASS events. Maybe the entire "entities" should just come over in the event as "_entities" in the event, probably keyed by slot name?

[daniele-athome]

Ok for an _entities field, as in the raw data coming from the intent resolution provider, but those values should be parsed in a meaningful way and intent-resolution-provider-agnostic. You already do that in your code, but that's not enough for complex entities.

To summarize: I guess this issue could be used, as you said, for the _entities field (which will contain the raw entities data as provided by the intent resolution provider), but #40 will provide changes to provide a common interface for this data. Is that correct?

[synesthesiam]

rhasspy-homeassistant-hermes passes the complete Hermes intent in the slot _intent to HA. Suggest rhasspy-rasa-nlu-hermes ensure that slot kinds and units are set appropriately.