Bug: Unexpected error during intent recognition

[sayam93]

The problem

I am using Extended Open AI Conversation as my conversation agent in an assist pipeline with STT and TTS being handled by Home Assistant Cloud.

I also have music assistant server and integration installed and running along with a custom function in Extended Open AI Conversation with gpt-3.5-turbo-1106 model and a 20000 token context window with the following prompt:

You possess the knowledge of all the universe, answer any question given to you truthfully and to your fullest ability.
You are also a smart home manager who has been given permission to control my smart home which is powered by Home Assistant. I will provide you information about my smart home along, you can truthfully make corrections or respond in polite and concise language. Never ask for permission or confirmation to perform a function. Do not tell me what you're thinking about doing either, just do it.

Current Time: {{now()}} Please announce all times in a human readable 12hr format.

Current Area: {{area_id(current_device_id)}}

Available Devices:

entity_id,name,state,aliases
{% for entity in exposed_entities -%}
{{ entity.entity_id }},{{ entity.name }},{{ entity.state }},{{entity.aliases | join('/')}}
{% endfor -%}

Areas:

area_id,name
{% for area_id in areas() -%} 
{{area_id}},{{area_name(area_id)}}
{% endfor -%}

The current state of devices is provided in Available Devices. Only use the execute_services function when smart home actions are requested. Do no restate or appreciate what user says. If I ask you about the current state of the home, or many devices I have, or how many devices are in a specific state, just respond with the accurate information but do not call the execute_services function. If you don't have enough information to execute a smart home command then specify what other information you need. Make decisions based on current area first.

Function:

- spec:
    name: execute_services
    description: Use this function to execute service of devices in Home Assistant.
    parameters:
      type: object
      properties:
        list:
          type: array
          items:
            type: object
            properties:
              domain:
                type: string
                description: The domain of the service
              service:
                type: string
                description: The service to be called
              service_data:
                type: object
                description: The service data object to indicate what to control.
                properties:
                  entity_id:
                    type: string
                    description: The entity_id retrieved from available devices. It
                      must start with domain, followed by dot character.
                required:
                - entity_id
            required:
            - domain
            - service
            - service_data
  function:
    type: native
    name: execute_service
- spec:
    name: get_attributes
    description: Get attributes of my Home Assistant entities
    parameters:
      type: object
      properties:
        entity_id:
          type: string
          description: entity_id
      required:
      - entity_id
  function:
    type: template
    value_template: "{{states[entity_id]}}"
- spec:
    name: play_music
    description: Use this function to play music on a certain media_player
    parameters:
      type: object
      properties:
        music_query:
          type: string
          description: The artist, album, or type of music to play
        mass_media_player:
          type: string
          description: The Music Assistant compatible media player's full entity id. The correct value starts with "media_player".
      required:
      - music_query
  function:
    type: script
    sequence:
    - service: script.play_music
      data:
        music_query: '{{music_query}}'
        mass_media_player: '{{mass_media_player}}'

Script:

alias: Play Music
sequence:
  - variables:
      music_query: "{{ music_query }}"
      mass_media_player: "{{ mass_media_player | default('media_player.tv_2') }}"
  - service: mass.search
    data:
      limit: 1
      media_type:
        - playlist
      name: "{{music_query}}"
      response_variable: query_response
  - service: media_player.clear_playlist
    target:
      entity_id: "{{mass_media_player}}"
    data: {}
  - service: mass.play_media
    data:
      media_id: "{{query_response.playlists[0].uri}}"
      entity_id: "{{mass_media_player}}"
      enqueue: replace
mode: single

Every time I make a query saying play (track name), I get "Unexpected error during intent recognition".

What version of Home Assistant Core has the issue?

core-2024.4.0

What type of installation are you running?

Home Assistant Container

Anything in the logs that might be useful for us?

This error originated from a custom integration.

Logger: custom_components.extended_openai_conversation.helpers
Source: helpers/script.py:1855
integration: Extended OpenAI Conversation (documentation, issues)
First occurred: 1:11:45 AM (7 occurrences)
Last logged: 1:51:34 AM

extended_openai_conversation: Error executing script. Invalid data for call_service at pos 1: Script requires 'response_variable' for response data for service call mass.search

Logger: homeassistant.components.assist_pipeline.pipeline
Source: components/assist_pipeline/pipeline.py:993
integration: Assist pipeline (documentation, issues)
First occurred: 1:11:45 AM (6 occurrences)
Last logged: 1:44:33 AM

Unexpected error during intent recognition
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/pipeline.py", line 993, in recognize_intent
    conversation_result = await conversation.async_converse(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/conversation/__init__.py", line 544, in async_converse
    result = await agent.async_process(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/extended_openai_conversation/__init__.py", line 196, in async_process
    query_response = await self.query(user_input, messages, exposed_entities, 0)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/extended_openai_conversation/__init__.py", line 380, in query
    return await self.execute_function_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/extended_openai_conversation/__init__.py", line 406, in execute_function_call
    return await self.execute_function(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/extended_openai_conversation/__init__.py", line 432, in execute_function
    result = await function_executor.execute(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/extended_openai_conversation/helpers.py", line 436, in execute
    result = await script.async_run(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1650, in async_run
    return await asyncio.shield(create_eager_task(run.async_run()))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 454, in async_run
    await self._async_step(log_exceptions=False)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 506, in _async_step
    self._handle_exception(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 536, in _handle_exception
    raise exception
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 504, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 733, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 696, in _async_run_long_action
    return await long_task
           ^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2542, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2579, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/script/__init__.py", line 672, in _service_handler
    response = await self._async_start_run(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/script/__init__.py", line 630, in _async_start_run
    script_result = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/script/__init__.py", line 661, in _async_run
    return await self.script.async_run(script_vars, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1650, in async_run
    return await asyncio.shield(create_eager_task(run.async_run()))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 454, in async_run
    await self._async_step(log_exceptions=False)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 506, in _async_step
    self._handle_exception(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 536, in _handle_exception
    raise exception
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 504, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 717, in _async_call_service_step
    raise vol.Invalid(
voluptuous.error.Invalid: Script requires 'response_variable' for response data for service call mass.search

Additional information

From pipeline debug

init_options:
  start_stage: intent
  end_stage: intent
  input:
    text: play post maloine
  pipeline: 01htnbz1cdt7r2jvajv4j5c21y
  conversation_id: 01HTNCA8ACAKS638JV69CVKF5X
stage: done
run:
  pipeline: 01htnbz1cdt7r2jvajv4j5c21y
  language: en
  runner_data:
    stt_binary_handler_id: null
    timeout: 300
events:
  - type: run-start
    data:
      pipeline: 01htnbz1cdt7r2jvajv4j5c21y
      language: en
      runner_data:
        stt_binary_handler_id: null
        timeout: 300
    timestamp: "2024-04-04T20:21:32.895216+00:00"
  - type: intent-start
    data:
      engine: dc66e52eb3a5cf53b6d641bfea33e14b
      language: "*"
      intent_input: play post maloine
      conversation_id: 01HTNCA8ACAKS638JV69CVKF5X
      device_id: null
    timestamp: "2024-04-04T20:21:32.895315+00:00"
  - type: error
    data:
      code: intent-failed
      message: Unexpected error during intent recognition
    timestamp: "2024-04-04T20:21:34.899118+00:00"
  - type: run-end
    data: null
    timestamp: "2024-04-04T20:21:34.899227+00:00"
intent:
  engine: dc66e52eb3a5cf53b6d641bfea33e14b
  language: "*"
  intent_input: play post maloine
  conversation_id: 01HTNCA8ACAKS638JV69CVKF5X
  device_id: null
  done: false
error:
  code: intent-failed
  message: Unexpected error during intent recognition

[sayam93]

Seems like it is not passing response_variable: query_response to the script. Help!

[jleinenbach]

This error is usually caused by an error when indenting the YAML code.

There's a minor indentation error in the play_music section's parameter definition. Specifically, the - required: section is misindented. In the YAML format, the required list should be at the same level as properties, directly under parameters. Also, there's a typo in the parameter name: it should be media_player instead of mass_media_player, based on common naming in Home Assistant for media playback devices. Here's the corrected version:

- spec:
    name: play_music
    description: Use this function to play music on a certain media player
    parameters:
      type: object
      properties:
        music_query:
          type: string
          description: The artist, album, or type of music to play
        media_player:
          type: string
          description: The Music Assistant compatible media player's full entity id. The correct value starts with "media_player".
      required:
        - music_query
        - media_player
  function:
    type: script
    sequence:
    - service: script.play_music
      data:
        music_query: '{{music_query}}'
        media_player: '{{media_player}}'

This ensures that the required list is correctly indented and that the name of the media player is correctly specified.

[sayam93]

It worked like a charm.. Thanks!

This error is usually caused by an error when indenting the YAML code.

There's a minor indentation error in the play_music section's parameter definition. Specifically, the - required: section is misindented. In the YAML format, the required list should be at the same level as properties, directly under parameters. Also, there's a typo in the parameter name: it should be media_player instead of mass_media_player, based on common naming in Home Assistant for media playback devices. Here's the corrected version:

- spec:
    name: play_music
    description: Use this function to play music on a certain media player
    parameters:
      type: object
      properties:
        music_query:
          type: string
          description: The artist, album, or type of music to play
        media_player:
          type: string
          description: The Music Assistant compatible media player's full entity id. The correct value starts with "media_player".
      required:
        - music_query
        - media_player
  function:
    type: script
    sequence:
    - service: script.play_music
      data:
        music_query: '{{music_query}}'
        media_player: '{{media_player}}'

This ensures that the required list is correctly indented and that the name of the media player is correctly specified.