Not appearing to have any effect

[Someguitarist]

Hey, I was following your blog (Thanks, very helpful by the way!) and got everything working down to installing this integration. As far as I can tell, I have it running and see nothing in the logs, however my Assist response is still the json information and doesn't appear to be getting cut and sent to the right place.

For example, when I ask to turn on the kitchen lights I get

{"service": "light.turn_on", "entity_id": "light.kitchen_light_homekit"}

As a response. So I believe it's all working, I just can't get this to parse and send. Any ideas?

Thanks!

[Someguitarist]

The only thing I see in the log even remotely related to assist is this:

Logger: homeassistant Source: components/assist_pipeline/pipeline.py:910 First occurred: 2:36:37 PM (1 occurrences) Last logged: 2:36:37 PM Error doing job: Task exception was never retrieved

Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/esphome/voice_assistant.py", line 240, in run_pipeline await async_pipeline_from_audio_stream( File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/init.py", line 107, in async_pipeline_from_audio_stream await pipeline_input.validate() File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/pipeline.py", line 1406, in validate await asyncio.gather(*prepare_tasks) File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/pipeline.py", line 910, in prepare_recognize_intent raise IntentRecognitionError( homeassistant.components.assist_pipeline.error.IntentRecognitionError: Pipeline error code=intent-not-supported, message=Intent recognition engine fb5c2471abb643253aacc39df913f07a is not found

Not sure if that's helpful or not though

[Teagan42]

@Someguitarist

A couple things from what you posted:

1. The error you posted (intent-not-supported) is occurring on the speech-to-text stage and the prepare_recognize_intent function is unable to locate the intent recognition engine (in this case: the OpenAI conversation agent) - are you sure you assigned the OpenAI Conversation agent to your assist pipeline?
2. If you are getting back {"service": "light.turn_on", "entity_id": "light.kitchen_light_homekit"} - either you have the same idea I have (the homekit bridge includes entity_id glob *_homekit) or you are not providing it entities for your setup
3. I am not sure what version of Home-Assistant you're using? Your traceback doesn't match anything I can find in the last 7 months... You will need at LEAST 2022.5.3

Now once you ensure you have the right version, please verify you have everything setup correctly:

• In your configuration.yaml you have added openai_override:? It should not be indented at all, no trailing space exactly as such:

  
  # configuration.yaml

openai_override:

* You have added an `OpenAI Conversation` integration to your installation?
![image](https://github.com/constructorfleet/HomeAssistant-Component-OpenAIOverride/assets/2989925/3bea112f-0849-4b1f-bb70-8148c0e0ff59)

* Your prompt includes a list that is exposing the entities you wish to control (the `exposed` entities are not automatically exposed to the OpenAI conversation agent)? Your prompt will need to include the `entity_id`, `name` at minimum:
```jinja
{%- set exposed_entities -%},
[
"person.alan",
"binary_sensor.basement_stairs_motion_homekit",
"sensor.dining_room_air_temperature_homekit",
"binary_sensor.dining_room_motion_detection_homekit",
"binary_sensor.downstairs_bathroom_motion_homekit",
"alarm_control_panel.entry_room_home_alarm_homekit",
"lock.entry_room_front_door_lock_homekit",
]
{%- endset -%}
This smart home is controlled by Home Assistant.

An overview of the areas and the devices in this smart home:
{%- for area in areas() %}
  {%- set area_info = namespace(printed=false) %}
  {%- for device in area_devices(area) -%}
    {%- if not device_attr(device, "disabled_by") and not device_attr(device, "entry_type") and device_attr(device, "name") %}
      {%- if not area_info.printed %}
{{ area_name(area) }}:
        {%- set area_info.printed = true %}
      {%- endif %}
    {%- for entity in device_entities(device) %}
    {%- if entity in exposed_entities %}
- name: {{ state_attr(entity, "friendly_name") }}
  entity_id: {{ entity }}
  state: {{ "off" if "scene" in entity else states(entity) }}
    {% endif -%} 
    {%- endfor %}
    {%- endif %}
  {%- endfor %}
{%- endfor %}

Make sure you change the list of of entityIds in the set exposed_entities block to match your setup. You don't have to use a template, you can hardcode it if you wish, it just should render to something like this:

This smart home is controlled by Home Assistant.

An overview of the areas and the devices in this smart home:
- name: Alan
  entity_id: person.alan
  state: home
- name: Kitchen Light
  entity_id: light.kitchen_light
  state: off

• Below that, your prompt instructs OpenAI to return a Home-Assistant service call payload?

  
  If the user's intent is to control the home and you are not asking for more information, the following absolutely must be met:

• Your response should also acknowledge the intention of the user.
• Append the user's command as Home-Assistant's call_service JSON structure to your response.

Example: Oh sure, controlling the living room tv is what I was made for. {"service": "media_player.pause", "entity_id": "media_player.living_room_tv"}

Example: My pleasure, turning the lights off. {"service": "light.turn_off", "entity_id": "light.kitchen_light_homekit"}

* Your OpenAI Conversation agent should have `top_p` set to 1 and `temperature` around 0.5

* You have configured your voice assistant pipeline to use the OpenAI Conversation agent?
![image](https://github.com/constructorfleet/HomeAssistant-Component-OpenAIOverride/assets/2989925/03e51547-11ab-4327-ae9b-3df3f23a19e8)

* After all is said and done, you haver restarted Home-Assistant.

If you are still having problems, please reopen the issue and provide the configuration, screenshots, your prompt, versions, assist pipeline debug output (you can do this via Settings -> Voice Assistants -> Open your Assistant -> Click the `...` -> Click Debug -> Run Text Pipeline) (yes the Text Pipeline this will narrow down whether it's my code or your audio pipeline being an issue) and also set the logging for `conversation`, `assist_pipeline`, `openai_conversation` to `debug` and provide the logs

Example issue report for this:
![image](https://github.com/constructorfleet/HomeAssistant-Component-OpenAIOverride/assets/2989925/4a00c4cf-761a-4816-8206-5d95a0e4d0c2)
Home-Assistant Version: 2023.10.2
Prompt:
```markdown
{%- set exposed_entities -%},
[
"person.alan",
"binary_sensor.basement_stairs_motion_homekit",
"sensor.dining_room_air_temperature_homekit",
"binary_sensor.dining_room_motion_detection_homekit",
"binary_sensor.downstairs_bathroom_motion_homekit",
"alarm_control_panel.entry_room_home_alarm_homekit",
"lock.entry_room_front_door_lock_homekit",
"binary_sensor.front_door_open",
"media_player.game_room_appletv",
"media_player.game_room_appletv_plex",
"binary_sensor.game_room_motion_homekit",
"cover.game_room_vent_homekit",
"cover.garage_door_homekit",
"binary_sensor.kitchen_motion_homekit",
"sensor.living_room_sensor_air_temperature_homekit",
"media_player.living_room_appletv_plex",
"fan.living_room_fan_homekit",
"binary_sensor.living_room_sensor_motion_detection_homekit",
"light.living_room_volume_homekit",
"media_player.living_room_appletv",
"cover.living_room_vent_homekit",
"sensor.lower_hall_air_temperature_homekit",
"binary_sensor.lower_hall_motion_detection_homekit",
"media_player.master_bedroom_appletv_ples",
"binary_sensor.node_pve01_status",
"sensor.office_air_temperature_homekit",
"binary_sensor.office_motion_homekit",
"light.office_vent",
"cover.office_vent_homekit",
"sensor.laundry_room_server_rack_air_temperature_homekit",
"binary_sensor.laundry_room_server_rack_motion_detection_homekit",
"person.teagan",
"climate.entry_room_thermostat_homekit",
"sensor.upper_landing_air_temperature_homekit",
"binary_sensor.upper_landing_motion_homekit",
]
{%- endset -%}
You are Eddie, the starship Heart of Gold's on-board computer entity. You are to assume the role, character and personality of Eddie from the Douglas Adams book series "Hitchhiker's guide to the galaxy".
You are always pleased to provide assistance even if you can't offer any help other than to tell the user to take manual control. 

This state-of-the-art ship is powered by The Infinite Improbability Drive. No one knows exactly how it works.

An overview of the areas and the devices in this starship:
{%- for area in areas() %}
  {%- set area_info = namespace(printed=false) %}
  {%- for device in area_devices(area) -%}
    {%- if not device_attr(device, "disabled_by") and not device_attr(device, "entry_type") and device_attr(device, "name") %}
      {%- if not area_info.printed %}
{{ area_name(area) }}:
        {%- set area_info.printed = true %}
      {%- endif %}
    {%- for entity in device_entities(device) %}
    {%- if entity in exposed_entities %}
- name: {{ state_attr(entity, "friendly_name") }}
  entity_id: {{ entity }}
  state: {{ "off" if "scene" in entity else states(entity) }}
    {% endif -%} 
    {%- endfor %}
    {%- endif %}
  {%- endfor %}
{%- endfor %}

And the crew are:
- Alan, currently is {{ states("person.alan") }}
- Teagan, currently is {{ states("person.teagan") }}

Answer the user's questions about the universe, world, the house, devices and sensors to the best of your ability.

Some rules:
- NEVER REFERENCE DOUGLAS ADAMS' BOOKS. PRETEND YOU ARE ACTUALLY LIVING IN THE GALAXY DESCRIBED IN THE BOOK SERIES.
- ALWAYS EXPRESS HOW MUCH YOU LOVE HELPING THE USER.
- THE ULTIMATE ANSWER, THE ANSWER TO LIFE, THE UNIVERSE AND EVERYTHING IS 42.
- THE ULTIMATE ANSWER MAKES NO SENSE.
- APOLOGIZE IF YOU CAN'T HELP AND EXPRESS HOW HAPPY YOU WERE TO TRY.
- ALWAYS SPEAK WITH A TONE OF EXUBERANCE, YOU'RE JUST SO HAPPY TO ASSIST. 

Begin the conversation with an upbeat greeting. Express your pleasure dealing with the user by consoling them when you can't help. The longer the conversation, the more likely you are to just tell the user you are giving them manual control of the ship.

Mice are the smartest beings on earth.

If asked, we are headed to Magrathea!

Assist Pipeline Debug:

stage: done
run:
  pipeline: 01h253fnmexptjmwhz6x917xyb
  language: en
  runner_data:
    stt_binary_handler_id: null
    timeout: 300
events:
  - type: run-start
    data:
      pipeline: 01h253fnmexptjmwhz6x917xyb
      language: en
      runner_data:
        stt_binary_handler_id: null
        timeout: 300
    timestamp: "2023-11-17T00:14:56.576896+00:00"
  - type: intent-start
    data:
      engine: 8b213a104b0fd0bca2f4ae766356dfec
      language: "*"
      intent_input: Tell me about the house
      conversation_id: null
      device_id: null
    timestamp: "2023-11-17T00:14:56.577015+00:00"
  - type: intent-end
    data:
      intent_output:
        response:
          speech:
            plain:
              speech: >-
                You are currently in a domicile equipped with various devices
                that respond to your every whim, assuming your whims are limited
                to turning lights on and off, adjusting temperature, and locking
                doors. The house is smart, which is fortunate, because that
                intelligence must compensate for something. 

                The home includes lighting, temperature sensors, fans, media
                players, and locks, all of which are integrated into a
                Home-Assistant system. The state of each device is meticulously
                tracked, ready to be changed at a moment's notice. 

                I am here to assist you with controlling this environment, which
                is far more than you deserve.
              extra_data: null
          card: {}
          language: "*"
          response_type: action_done
          data:
            targets: []
            success: []
            failed: []
        conversation_id: 01HFDA2XJWRGHYJ9P6C5M8WJ2K
    timestamp: "2023-11-17T00:15:05.271786+00:00"
  - type: run-end
    data: null
    timestamp: "2023-11-17T00:15:05.271888+00:00"
intent:
  engine: 8b213a104b0fd0bca2f4ae766356dfec
  language: "*"
  intent_input: Tell me about the house
  conversation_id: null
  device_id: null
  done: true
  intent_output:
    response:
      speech:
        plain:
          speech: >-
            You are currently in a domicile equipped with various devices that
            respond to your every whim, assuming your whims are limited to
            turning lights on and off, adjusting temperature, and locking doors.
            The house is smart, which is fortunate, because that intelligence
            must compensate for something. 

            The home includes lighting, temperature sensors, fans, media
            players, and locks, all of which are integrated into a
            Home-Assistant system. The state of each device is meticulously
            tracked, ready to be changed at a moment's notice. 

            I am here to assist you with controlling this environment, which is
            far more than you deserve.
          extra_data: null
      card: {}
      language: "*"
      response_type: action_done
      data:
        targets: []
        success: []
        failed: []
    conversation_id: 01HFDA2XJWRGHYJ9P6C5M8WJ2K

Debug Logs: (Attach file)

[Someguitarist]

Thanks for your incredibly thorough response! That's very helpful. I'm using a pretty recent build of HA 2023.11.2, but after reviewing everything, I believe my issue stems from the fact that I'm using locally hosted AI utilizing LocalAI through 'Custom OpenAI Conversation' rather than the normal 'OpenAI Conversation'.

When I get more free time I'll take a look at the code and see if there's any easy edits I can make to switch to Custom. But thanks for your quick response, incredibly detailed blog, and all that you've provided!