search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
78.35k stars 33.46k forks source link

HassLightSet color parameter from Google Generative AI #128314

Closed zed0 closed 6 months ago

zed0 commented 6 months ago

The problem

When asking a voice assistant using Google Generative AI to change the color of a light the voice assistant refuses citing an IntentUnexpectedError.

The logs (below) show that utils.color_name_to_rgb was called with a color_name parameter which is a RGBColor when the type hinting suggests that it should be a str.

I was able to work around this problem by adding the following lines at the start of color_name_to_rgb, but realise that this is just a hack which covers up the actual source of the issue:

if isinstance(color_name, RGBColor):
    return color_name

Using the Text Pipeline produces the following logs (relevant snippet here, full logs attached below):

2024-10-13 21:26:41.301 DEBUG (MainThread) [homeassistant.components.conversation.agent_manager] Processing in *: Set living room lights to red
2024-10-13 21:26:41.301 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: entity_id=conversation.google_generative_ai_2, old_state=<state conversation.google_generative_ai_2=2024-10-13T19:22:57.622570+00:00; friendly_name=Google Generative AI for conversations, supported_features=1 @ 2024-10-13T21:22:57.622598+02:00>, new_state=<state conversation.google_generative_ai_2=2024-10-13T19:26:41.301677+00:00; friendly_name=Google Generative AI for conversations, supported_features=1 @ 2024-10-13T21:26:41.301705+02:00>>
2024-10-13 21:26:41.315 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Input: 'Set living room lights to red' with history: []
2024-10-13 21:26:41.846 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Response: [function_call {
  name: "HassLightSet"
  args {
    fields {
      key: "color"
      value {
        string_value: "red"
      }
    }
    fields {
      key: "area"
      value {
        string_value: "Living Room"
      }
    }
  }
}
]
2024-10-13 21:26:41.847 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Tool call: HassLightSet({'color': 'red', 'area': 'Living Room'})
2024-10-13 21:26:41.847 INFO (MainThread) [homeassistant.helpers.intent] Triggering intent handler <ServiceIntentHandler - HassLightSet>
2024-10-13 21:26:41.847 ERROR (MainThread) [homeassistant.helpers.intent] Error handling HassLightSet
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/intent.py", line 142, in async_handle
    result = await handler.async_handle(intent)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/intent.py", line 917, in async_handle
    slots = self.async_validate_slots(intent_obj.slots)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/intent.py", line 770, in async_validate_slots
    return self._slot_schema(slots)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 205, in __call__
    return self._compiled([], data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 549, in validate_dict
    return base_validate(path, data.items(), out)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 330, in validate_mapping
    cval = cvalue(key_path, value)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 779, in validate_callable
    return schema(data)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 205, in __call__
    return self._compiled([], data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 549, in validate_dict
    return base_validate(path, data.items(), out)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 330, in validate_mapping
    cval = cvalue(key_path, value)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 779, in validate_callable
    return schema(data)
           ^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/color.py", line 204, in color_name_to_rgb
    hex_value = COLORS.get(color_name.replace(" ", "").lower())
                           ^^^^^^^^^^^^^^^^^^
AttributeError: 'RGBColor' object has no attribute 'replace'. Did you mean: '_replace'?
2024-10-13 21:26:41.850 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Tool response: {'error': 'IntentUnexpectedError', 'error_text': 'Error handling HassLightSet'}
2024-10-13 21:26:41.954 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event state_reported[L]: entity_id=sensor.jellyfin, old_last_reported=2024-10-13T21:26:31.953493+02:00, new_state=<state sensor.jellyfin=0; unit_of_measurement=clients, friendly_name=jellyfin Active clients @ 2024-10-13T21:21:12.634873+02:00>>
2024-10-13 21:26:42.444 DEBUG (MainThread) [homeassistant.components.google_generative_ai_conversation] Response: [text: "I am sorry, I cannot fulfill this request. I am unable to set the color of the lights. \n"
]

What version of Home Assistant Core has the issue?

core-2024.10.2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Google Generative AI

Link to integration documentation on our website

https://www.home-assistant.io/integrations/google_generative_ai_conversation

Diagnostics information

home-assistant_google_generative_ai_conversation_2024-10-13T19-27-27.409Z.log

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: homeassistant.helpers.intent
Source: helpers/intent.py:142
First occurred: 21:22:33 (1 occurrences)
Last logged: 21:22:33

Error handling HassLightSet
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/intent.py", line 142, in async_handle
    result = await handler.async_handle(intent)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/intent.py", line 917, in async_handle
    slots = self.async_validate_slots(intent_obj.slots)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/intent.py", line 770, in async_validate_slots
    return self._slot_schema(slots)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 205, in __call__
    return self._compiled([], data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 549, in validate_dict
    return base_validate(path, data.items(), out)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 330, in validate_mapping
    cval = cvalue(key_path, value)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 779, in validate_callable
    return schema(data)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 205, in __call__
    return self._compiled([], data)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 549, in validate_dict
    return base_validate(path, data.items(), out)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 330, in validate_mapping
    cval = cvalue(key_path, value)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/voluptuous/schema_builder.py", line 779, in validate_callable
    return schema(data)
           ^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/color.py", line 204, in color_name_to_rgb
    hex_value = COLORS.get(color_name.replace(" ", "").lower())
                           ^^^^^^^^^^^^^^^^^^
AttributeError: 'RGBColor' object has no attribute 'replace'. Did you mean: '_replace'?


### Additional information

_No response_
home-assistant[bot] commented 6 months ago

Hey there @tronikos, mind taking a look at this issue as it has been labeled with an integration (google_generative_ai_conversation) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `google_generative_ai_conversation` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign google_generative_ai_conversation` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

(message by CodeOwnersMention)

google_generative_ai_conversation documentation google_generative_ai_conversation source (message by IssueLinks)

alcoclown commented 6 months ago

I have exactly the same problem with the OpenAI integration.

tronikos commented 6 months ago

@Shulyaka this is caused by https://github.com/home-assistant/core/pull/123621 I've confirmed that reverting it locally fixes the issue.

Shulyaka commented 6 months ago

The problem must be deeper, I will look into this

home-assistant[bot] commented 6 months ago

Hey there @balloob, mind taking a look at this issue as it has been labeled with an integration (openai_conversation) you are listed as a code owner for? Thanks!

Code owner commands Code owners of `openai_conversation` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign openai_conversation` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

(message by CodeOwnersMention)

openai_conversation documentation openai_conversation source (message by IssueLinks)