ChatGPT as Assist Conversation Agent via OpenAI Conversation Integration Gives "intent-failed" Error

FortranFour commented 3 months ago

The problem

I have used the OpenAI Conversation to configure a ChatGPT conversation agent. If I configure "Control Home Assistant" for "No control" it can answer general questions (e.g. "Who is President?"). If I ask it to turn on the kitchen light, it responds, "Please edit the AI configuration and allow access..." When I change the "Control Home Assistant" to "Assist", the response is "Unexpected error during intent recognition" in a red bubble, regardless of whether it is a general question (e.g. "Who is president?") or a command (e.g. "Turn on the kitchen light").

What version of Home Assistant Core has the issue?

core-2024.6.1

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

OpenAI Conversation

Link to integration documentation on our website

https://www.home-assistant.io/integrations/openai_conversation/

Diagnostics information

Source: components/assist_pipeline/pipeline.py:994
integration: Assist pipeline (documentation, issues)
First occurred: 10:04:44 AM (3 occurrences)
Last logged: 3:39:08 PM

Unexpected error during intent recognition
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/pipeline.py", line 994, in recognize_intent
    conversation_result = await conversation.async_converse(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/conversation/agent_manager.py", line 108, in async_converse
    result = await method(conversation_input)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/conversation/entity.py", line 45, in internal_async_process
    return await self.async_process(user_input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/openai_conversation/conversation.py", line 128, in async_process
    llm_api = await llm.async_get_api(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/llm.py", line 86, in async_get_api
    return await apis[api_id].async_get_api_instance(llm_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/llm.py", line 276, in async_get_api_instance
    api_prompt=self._async_get_api_prompt(llm_context, exposed_entities),
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/llm.py", line 335, in _async_get_api_prompt
    prompt.append(yaml.dump(exposed_entities))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/yaml/dumper.py", line 23, in dump
    return yaml.dump(
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/__init__.py", line 253, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/__init__.py", line 241, in dump_all
    dumper.represent(data)
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 27, in represent
    node = self.represent_data(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 58, in represent_data
    node = self.yaml_representers[None](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 231, in represent_undefined
    raise RepresenterError("cannot represent an object", data)
yaml.representer.RepresenterError: ('cannot represent an object', 62.1)

Example YAML snippet

No response

Anything in the logs that might be useful for us?

#Debug Assistant from Assist:

stage: done
run:
  pipeline: 01hzwb3jvrq8a1p8gzpqmex9wg
  language: en
  runner_data:
    stt_binary_handler_id: null
    timeout: 300
events:
  - type: run-start
    data:
      pipeline: 01hzwb3jvrq8a1p8gzpqmex9wg
      language: en
      runner_data:
        stt_binary_handler_id: null
        timeout: 300
    timestamp: "2024-06-08T22:45:44.527733+00:00"
  - type: intent-start
    data:
      engine: conversation.chatgpt_2
      language: "*"
      intent_input: Who is President?
      conversation_id: null
      device_id: null
    timestamp: "2024-06-08T22:45:44.527817+00:00"
  - type: error
    data:
      code: intent-failed
      message: Unexpected error during intent recognition
    timestamp: "2024-06-08T22:45:44.548145+00:00"
  - type: run-end
    data: null
    timestamp: "2024-06-08T22:45:44.548188+00:00"
intent:
  engine: conversation.chatgpt_2
  language: "*"
  intent_input: Who is President?
  conversation_id: null
  device_id: null
  done: false
error:
  code: intent-failed
  message: Unexpected error during intent recognition

Additional information

No response

home-assistant[bot] commented 3 months ago

Hey there @balloob, mind taking a look at this issue as it has been labeled with an integration (openai_conversation) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `openai_conversation` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign openai_conversation` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

openai_conversation documentation openai_conversation source _{^{(message by IssueLinks)}}

Spegeli commented 3 months ago

You are not alone with that issue ;-)

azdlowry commented 3 months ago

It seems to be intermittent for me. Works for a while then stops working. Maybe some entity state that is failing to be to added to prompt. I shall try to gather more info.

balloob commented 3 months ago

This looks like one of your entities has an attribute that cannot be represented in YAML. Can you see what entity that you have exposed has an attribute with a value 61.2 ?

FortranFour commented 3 months ago

Awesome, thanks, balloob! That was it, a beach temperature sensor (in Fahrenheit) derived from a hacky NOAA integration (the beach closest to where you wrote the first lines of code, by the way). I should have picked this up from the log. Interestingly, equivalent Assist Conversation Agents made using the custom component, Extended OpenAI Conversation, did not have this issue.

Spegeli commented 3 months ago

This looks like one of your entities has an attribute that cannot be represented in YAML. Can you see what entity that you have exposed has an attribute with a value 61.2 ?

Ah good to know. For me its currently the "79" i see in the logs, only issue: to man entities with a attribute of 79 g But now i know where i have to log.

And yeah with the Extended OpenAI this issue does not exists.

jolbi commented 3 months ago

In my case, the value is "36". I went through all the entities and there were only 3 entities with full match attribute "36". I removed them from assist, but the error message stays the same. Could the value be only a part of an attribute?

It is also intermittent for me. After a restart, it usually works for some time but then stops.

Liquidmasl commented 3 months ago

with the new update i tried connecting to openai as well. but sadly even with one single light exposed i get this error... not sure whatsup there

.. it works with a switch entity though

ahaverty commented 2 months ago

I'm intermittently seeing the same on v2024.6.4, and still seeing the same issue when I manually apply @balloob's fix for decimals.

Edit: I've been running my PR for 3 days now and haven't hit this issue since: https://github.com/home-assistant/core/pull/121037 Hopefully a maintainer can review soon 🙏

websocket_api script: Error executing script. Unexpected error for call_service at pos 1: ('cannot represent an object', 175)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 526, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 763, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 726, in _async_run_long_action
    return await long_task
           ^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2741, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2784, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/conversation/__init__.py", line 208, in handle_process
    result = await async_converse(
             ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/conversation/agent_manager.py", line 108, in async_converse
    result = await method(conversation_input)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/conversation/entity.py", line 45, in internal_async_process
    return await self.async_process(user_input)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/google_generative_ai_conversation/conversation.py", line 179, in async_process
    llm_api = await llm.async_get_api(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/llm.py", line 89, in async_get_api
    return await apis[api_id].async_get_api_instance(llm_context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/llm.py", line 283, in async_get_api_instance
    api_prompt=self._async_get_api_prompt(llm_context, exposed_entities),
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/llm.py", line 342, in _async_get_api_prompt
    prompt.append(yaml.dump(exposed_entities))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/yaml/dumper.py", line 23, in dump
    return yaml.dump(
           ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/__init__.py", line 253, in dump
    return dump_all([data], stream, Dumper=Dumper, **kwds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/__init__.py", line 241, in dump_all
    dumper.represent(data)
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 27, in represent
    node = self.represent_data(data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 48, in represent_data
    node = self.yaml_representers[data_types[0]](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 207, in represent_dict
    return self.represent_mapping('tag:yaml.org,2002:map', data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 118, in represent_mapping
    node_value = self.represent_data(item_value)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 58, in represent_data
    node = self.yaml_representers[None](self, data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/yaml/representer.py", line 231, in represent_undefined
    raise RepresenterError("cannot represent an object", data)
yaml.representer.RepresenterError: ('cannot represent an object', 175)

home-assistant / core