Intent clashes with Home Assistant skill

[scottsweb]

I am testing this with the Home Assistant skill on the commonIot branch and things are working quite well. Unfortunately there are certain devices that are not picked up correctly. A good example is a Home Assistant group/switch called Lounge Lights. This skill seems to attempt to interpret the event as just 'lights'. From the logs:

{"type": "recognizer_loop:utterance", "data": {"utterances": ["turn on the lounge lights"], "lang": "en-US", "session": "ea17580a-d9d3-4843-ac6f-dc3ea03cc34a"}, "context": {"client_name": "mycroft_listener", "ident": "1566331094.035573-1507550021"}}
{"type": "skill-iot-control.mycroftai:IoTRequestWithEntityOrThing", "data": {"intent_type": "skill-iot-control.mycroftai:IoTRequestWithEntityOrThing", "skill_iot_control_mycroftaiLIGHT": "light", "skill_iot_control_mycroftaiON": "on", "target": null, "confidence": 0.2916666666666667, "__tags__": [{"skill_iot_control_mycroftaiLIGHT": [{"match": "light", "key": "light", "start_token": 4, "entities": [{"key": "light", "match": "light", "data": [["light", "skill_iot_control_mycroftaiLIGHT"]], "confidence": 1.0}], "end_token": 4, "from_context": false}], "skill_iot_control_mycroftaiON": [{"match": "on", "key": "on", "start_token": 1, "entities": [{"key": "on", "match": "on", "data": [["on", "skill_iot_control_mycroftaiON"], ["on", "skill_iot_control_mycroftaiPOWERED"]], "confidence": 1.0}], "end_token": 1, "from_context": false}]}], "utterance": "turn on the lounge lights"}, "context": {"client_name": "mycroft_listener", "ident": "1566331094.035573-1507550021", "target": null}}
{"type": "mycroft.skill.handler.start", "data": {"name": "SkillIoTControl._handle_iot_request"}, "context": {"client_name": "mycroft_listener", "ident": "1566331094.035573-1507550021", "target": null}}
{"type": "iot:trigger", "data": {"intent_type": "skill-iot-control.mycroftai:IoTRequestWithEntityOrThing", "target": null, "confidence": 0.2916666666666667, "__tags__": [{"skill_iot_control_mycroftaiLIGHT": [{"match": "light", "key": "light", "start_token": 4, "entities": [{"key": "light", "match": "light", "data": [["light", "skill_iot_control_mycroftaiLIGHT"]], "confidence": 1.0}], "end_token": 4, "from_context": false}], "skill_iot_control_mycroftaiON": [{"match": "on", "key": "on", "start_token": 1, "entities": [{"key": "on", "match": "on", "data": [["on", "skill_iot_control_mycroftaiON"], ["on", "skill_iot_control_mycroftaiPOWERED"]], "confidence": 1.0}], "end_token": 1, "from_context": false}]}], "utterance": "turn on the lounge lights", "LIGHT": "light", "ON": "on", "iot_request_id": "abbaaf15-4e5d-4990-be4d-6c2269d9cc1b", "IoTRequest": {"action": "ON", "thing": "LIGHT", "attribute": null, "entity": null, "scene": null, "value": null, "state": null}}, "context": {}}
{"type": "mycroft.scheduler.schedule_event", "data": {"time": 1566331105.0, "event": "skill-iot-control.mycroftai:DeleteRequest", "repeat": null, "data": {"iot_request_id": "abbaaf15-4e5d-4990-be4d-6c2269d9cc1b"}}, "context": {}}
{"type": "mycroft.scheduler.schedule_event", "data": {"time": 1566331096.0, "event": "skill-iot-control.mycroftai:RunIotRequest", "repeat": null, "data": {"iot_request_id": "abbaaf15-4e5d-4990-be4d-6c2269d9cc1b"}}, "context": {}}
{"type": "mycroft.skill.handler.complete", "data": {"name": "SkillIoTControl._handle_iot_request"}, "context": {"client_name": "mycroft_listener", "ident": "1566331094.035573-1507550021", "target": null}}
{"type": "iot:trigger.response", "data": {"intent_type": "skill-iot-control.mycroftai:IoTRequestWithEntityOrThing", "target": null, "confidence": 0.2916666666666667, "__tags__": [{"skill_iot_control_mycroftaiLIGHT": [{"match": "light", "key": "light", "start_token": 4, "entities": [{"key": "light", "match": "light", "data": [["light", "skill_iot_control_mycroftaiLIGHT"]], "confidence": 1.0}], "end_token": 4, "from_context": false}], "skill_iot_control_mycroftaiON": [{"match": "on", "key": "on", "start_token": 1, "entities": [{"key": "on", "match": "on", "data": [["on", "skill_iot_control_mycroftaiON"], ["on", "skill_iot_control_mycroftaiPOWERED"]], "confidence": 1.0}], "end_token": 1, "from_context": false}]}], "utterance": "turn on the lounge lights", "LIGHT": "light", "ON": "on", "iot_request_id": "abbaaf15-4e5d-4990-be4d-6c2269d9cc1b", "IoTRequest": {"action": "ON", "thing": "LIGHT", "attribute": null, "entity": null, "scene": null, "value": null, "state": null}, "skill_id": "homeassistant.mycroftai", "callback_data": {"domain": "light", "service": "turn_on", "states": [{}]}}, "context": {"target": null}}
{"type": "skill-iot-control.mycroftai:RunIotRequest", "data": {"iot_request_id": "abbaaf15-4e5d-4990-be4d-6c2269d9cc1b"}, "context": {}}
{"type": "iot:run.homeassistant.mycroftai", "data": {"intent_type": "skill-iot-control.mycroftai:IoTRequestWithEntityOrThing", "target": null, "confidence": 0.2916666666666667, "__tags__": [{"skill_iot_control_mycroftaiLIGHT": [{"match": "light", "key": "light", "start_token": 4, "entities": [{"key": "light", "match": "light", "data": [["light", "skill_iot_control_mycroftaiLIGHT"]], "confidence": 1.0}], "end_token": 4, "from_context": false}], "skill_iot_control_mycroftaiON": [{"match": "on", "key": "on", "start_token": 1, "entities": [{"key": "on", "match": "on", "data": [["on", "skill_iot_control_mycroftaiON"], ["on", "skill_iot_control_mycroftaiPOWERED"]], "confidence": 1.0}], "end_token": 1, "from_context": false}]}], "utterance": "turn on the lounge lights", "LIGHT": "light", "ON": "on", "iot_request_id": "abbaaf15-4e5d-4990-be4d-6c2269d9cc1b", "IoTRequest": {"action": "ON", "thing": "LIGHT", "attribute": null, "entity": null, "scene": null, "value": null, "state": null}, "skill_id": "homeassistant.mycroftai", "callback_data": {"domain": "light", "service": "turn_on", "states": [{}]}}, "context": {}}
{"type": "mycroft.scheduler.schedule_event", "data": {"time": 1566331097.0, "event": "skill-iot-control.mycroftai:SpeakOrAcknowledge", "repeat": null, "data": {"iot_request_id": "abbaaf15-4e5d-4990-be4d-6c2269d9cc1b"}}, "context": {}}
{"type": "skill-iot-control.mycroftai:SpeakOrAcknowledge", "data": {"iot_request_id": "abbaaf15-4e5d-4990-be4d-6c2269d9cc1b"}, "context": {}}

Home Assistant itself is not overly happy with the request:

Not passing an entity ID to a service to target all entities is deprecated. Update your call to light.turn_on to be instead: entity_id: all

It seems to work if I ask turn on lounge (without the lights on the end), but it seems to use older code in the Home Assistant skill - the command is confirmed with a reply, turned lounge lights on rather than the new confirmation beep.

{"type": "recognizer_loop:wakeword", "data": {"utterance": "hey mycroft", "session": "2ab76e24-afc3-4fb2-97a4-b48a6e385a02"}, "context": {}}
{"type": "recognizer_loop:utterance", "data": {"utterances": ["turn off lounge"], "lang": "en-US", "session": "2ab76e24-afc3-4fb2-97a4-b48a6e385a02"}, "context": {"client_name": "mycroft_listener", "ident": "1566333784.4796295221831097"}}
{"type": "skill.converse.request", "data": {"skill_id": "homeassistant.mycroftai", "utterances": ["turn off lounge"], "lang": "en-US"}, "context": {}}
{"type": "skill.converse.response", "data": {"skill_id": "homeassistant.mycroftai", "result": false}, "context": {}}
{"type": "skill.converse.request", "data": {"skill_id": "fallback-unknown.mycroftai", "utterances": ["turn off lounge"], "lang": "en-US"}, "context": {}}
{"type": "skill.converse.response", "data": {"skill_id": "fallback-unknown.mycroftai", "result": false}, "context": {}}
{"type": "intent_failure", "data": {"utterance": "turn off lounge", "norm_utt": "turn off lounge", "lang": "en-US"}, "context": {"client_name": "mycroft_listener", "ident": "1566333784.4796295221831097"}}
{"type": "mycroft.skill.handler.start", "data": {"handler": "fallback"}, "context": {"client_name": "mycroft_listener", "ident": "1566333784.4796295221831097"}}
{"type": "speak", "data": {"utterance": "Turned Lounge Lights off", "expect_response": false}, "context": {"client_name": "mycroft_listener", "ident": "1566333784.4796295221831097"}}

Although this worked on this occasion, generally when mycroft falls back to fallback-unknown.mycroftai for Home Assistant, the wrong device gets switched. For example turn on hot water, gets picked up as turned on hot water cancel.

This didn't used to be an issue on the old 18 branch of Mycroft with the older Home Assistant skill.

[krisgesling]

Hey Scott, thanks for the clear and detailed outline.

We don't currently have a team member actively working on this Skill as we've got the whole team focused on the Mark II, but this will be useful for whoever picks it up. Whether that's a team member or someone from the broader Community.

One small question, when you said:

For example turn on hot water, gets picked up as turned on hot water cancel.

Do you mean the STT transcribes the utterance incorrectly?

[scottsweb]

Thanks for the reply @krisgesling. From the logs it looks like the STT transcribes correctly, it is just the skills interpretation that is wrong - or at least a little unexpected. I have noticed in that second example I can say turn on hot water 30 mins and it will find the correct switch. Looking at my config there are two names configured for that particular switch:

https://github.com/scottsweb/ham/blob/master/homeassistant/customize.yaml#L150-L151

It used to be the case that the emulated hue name was used (old home assistant skill), now it seems to be the friendly name.

[ChristopherRogers1991]

It looks like the issue here is that 'lounge' is not recognized as an entity. We can see that from the intent type: IoTRequestWithEntityOrThing. If lounge was an entity, the intent would be an IoTRequestWithEntityAndThing - note the And instead of the Or (the 'Thing' is the light, which it picks up correctly). See https://github.com/MycroftAI/skill-iot-control/blob/master/__init__.py#L107-L122.

The is further evidenced by the behavior when you say "Turn off lounge" - it only works because of the fallback. Adapt doesn't even send it to the IoT control skill, because "turn off" doesn't match without an entity or thing included in the utterance, and 'lounge' isn't recognized.

This issue should be closed here, and opened on the Home Assistant skill - if Home Assistant registers "lounge" as an entity, this would work (you could test this by hard coding ["lounge"] as the result of the get_entities method in the Home Assistant skill). My guess would be the Home Assistant API changed and/or was augmented, and the skill just needs to be updated to account for that and pick up all the entities.

It would also need to be updated to account for:

Not passing an entity ID to a service to target all entities is deprecated. Update your call to light.turn_on to be instead: entity_id: all