Error on HA 2023.9.1

[tanker11]

Hello, it is a very nice work and I like the result a lot! It gave me a lot of inputs to "teach" my assistant understand the states of the house. I cannot however use the controls, as I have the following error in the assistant dialog: "Unexpected error during intent recognition". And seemingly during start there is an error related to openai_override: [homeassistant.helpers.integration_platform] Unexpected error importing openai_override/diagnostics.py

Is it something related to the HA version?

[ltjessem]

Was just about to make an issue about this. Seems to be related to this: https://developers.home-assistant.io/blog/2023/06/14/service-calls/

Removing the limit argument @ line 63 in __init__.py seems to fix it

[tanker11]

Thanks a lot. I removed the argument as per the following, restarted HA, but no change. Same error within Assist and erro in the log as well.

[ltjessem]

I did get that error once after making the change, but I haven't been able to reproduce it while having debug logging on since then, typical. Paste your prompt into Developer tools > Template and make sure your entities are looking ok. I found I had to tweak the example from the website, both because I don't really use "Areas", and because that example didn't actually pass on the entity id the AI needs to actually control them.

This is my simplified version:

{%- set exposed_entities = [
    "sensor.living_room_temperature",
    "light.dining_room"
] -%}

{%- for entity_id in exposed_entities %}
{{ state_attr(entity_id, "friendly_name") }} ({{ entity_id }}) = {{ states(entity_id) }}
{%- endfor %}

Which returns data which seems a bit easier for the AI to ingest:

Living room Temperature (sensor.living_room_temperature) = 25.3
Dining room light (light.dining_room) = off

[tanker11]

Mine was different a bit, but giving it a bit of patience, it started working. I also used your template, and seems more stable in selecting a correct entity. So I think the error appears indeed when the entity (or the service) was not correctly recognised. FYI, I also added an instruction for the dimmers: [...] * Append the user's command as Home-Assistant's call_service JSON structure to your response. In case of brightness, the value should go to the "data:" section.

But if the cause is an incorrect recognition, would it be possible to log the incoming text from OpenAI before the processing? In that case I could fine tune the behavior based on the response. Because as of today, if there is an error, I cannot see the response even from the Assis->Debug section.

[ltjessem]

Debugging works for me, I turn it on from here:

This is the relevant part of the log I got out when testing:

2023-09-27 07:40:05.869 DEBUG (MainThread) [homeassistant.components.openai_conversation] Prompt for gpt-3.5-turbo: [{'role': 'system', 'content': '<my prompt>'}, {'role': 'user', 'content': "turn off all the lights, you're wasting electricity "}]
2023-09-27 07:40:11.772 DEBUG (MainThread) [homeassistant.components.openai_conversation] Response {
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Damn, someone's concerned about the environment. I got you covered, bruh.\n{\"service\": \"light.turn_off\", \"entity_id\": \"light.floor_lamp\"}\n{\"service\": \"light.turn_off\", \"entity_id\": \"light.silicon_labs_ezsp_living_room_zha_group_0x0002\"}\n{\"service\": \"light.turn_off\", \"entity_id\": \"light.dining_room\"}\n{\"service\": \"light.turn_off\", \"entity_id\": \"light.boys\"}",
        "role": "assistant"
      }
    }
  ],
  "created": 1695793206,
  "id": "chatcmpl",
  "model": "gpt-3.5-turbo-0613",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 123,
    "prompt_tokens": 675,
    "total_tokens": 798
  }
}

Quite happy so far with how it handles multiple actions at a time. It still sometimes tries to interact with hallucinated entities but I saw a big improvement after setting Temperature to 0.5 (from 0.7) and Top P to 0.9 (from 1), and making sure to mention several times to only interact with entities he knows about. Still not hard to convince him to try to control other things though. I think an even lower temperature would help, but then it's less of a "character" which is what I'm going for.

Here's the status of the house and the only devices Davis can control. The states of these devices must be checked before making a response:
{%- for entity_id in exposed_entities %}
{{ state_attr(entity_id, "friendly_name") }} ({{ entity_id }}) = {{ states(entity_id) }}
{%- endfor %}
If they user asks about the state of something, Davis responds using this information. 
Davis does not ever interact with any entity not on this list, and instead berates the user. 

[tanker11]

Thanks for the debug hint, I get it now to start from the integration config. Also good that you suggest some Heat and Top P values, I realized it is worth playing them.

Now having the log in front of me I realized something strange. When I get the "Unexpected error during intent recognition" error, and see what was the response of OpenAI it seems ok, no hallucination. Even when I copy+paste the response content into the Developer Tools->Services and call the service, the command is successful, and it makes the requested control. What I see more in the log is the following:

2023-09-28 08:20:04.166 ERROR (MainThread) [homeassistant.components.assist_pipeline.pipeline] Unexpected error during intent recognition Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/pipeline.py", line 774, in recognize_intent conversation_result = await conversation.async_converse( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/conversation/init.py", line 467, in async_converse result = await agent.async_process( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/HomeAssistant-Component-OpenAIOverride-main/init.py", line 53, in async_process service_call = json.loads(segment) ^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/init.py", line 346, in loads return _default_decoder.decode(s) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) ^^^^^^^^^^^^^^^^^^^^^^ json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes: line 1 column 2 (char 1)

Is it something that can be fixed in the code, or I need to amend the instructions I give to "Davis"?

ltjessem @.***> ezt írta (időpont: 2023. szept. 27., Sze, 8:24):

Debugging works for me, I turn it on from here: [image: image] https://user-images.githubusercontent.com/10084953/270873991-be3dcdef-4d3b-4051-b105-78ab26ffc955.png

This is the relevant part of the log I got out when testing:

2023-09-27 07:40:05.869 DEBUG (MainThread) [homeassistant.components.openai_conversation] Prompt for gpt-3.5-turbo: [{'role': 'system', 'content': ''}, {'role': 'user', 'content': "turn off all the lights, you're wasting electricity "}] 2023-09-27 07:40:11.772 DEBUG (MainThread) [homeassistant.components.openai_conversation] Response { "choices": [ { "finish_reason": "stop", "index": 0, "message": { "content": "Damn, someone's concerned about the environment. I got you covered, bruh.\n{\"service\": \"light.turn_off\", \"entity_id\": \"light.floor_lamp\"}\n{\"service\": \"light.turn_off\", \"entity_id\": \"light.silicon_labs_ezsp_living_room_zha_group_0x0002\"}\n{\"service\": \"light.turn_off\", \"entity_id\": \"light.dining_room\"}\n{\"service\": \"light.turn_off\", \"entity_id\": \"light.boys\"}", "role": "assistant" } } ], "created": 1695793206, "id": "chatcmpl", "model": "gpt-3.5-turbo-0613", "object": "chat.completion", "usage": { "completion_tokens": 123, "prompt_tokens": 675, "total_tokens": 798 } }

Quite happy so far with how it handles multiple actions at a time. It still sometimes tries to interact with hallucinated entities but I saw a big improvement after setting Temperature to 0.5 (from 0.7) and Top P to 0.9 (from 1), and making sure to mention several times to only interact with entities he knows about. Still not hard to convince him to try to control other things though. I think an even lower temperature would help, but then it's less of a "character" which is what I'm going for.

Here's the status of the house and the only devices Davis can control. The states of these devices must be checked before making a response: {%- for entity_id in exposed_entities %} {{ state_attr(entity_id, "friendly_name") }} ({{ entity_id }}) = {{ states(entity_id) }} {%- endfor %} If they user asks about the state of something, Davis responds using this information. Davis does not ever interact with any entity not on this list, and instead berates the user.

— Reply to this email directly, view it on GitHub https://github.com/constructorfleet/HomeAssistant-Component-OpenAIOverride/issues/1#issuecomment-1736776053, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVM6O5RAKKJ6VDKP4QU6RLX4PBCPANCNFSM6AAAAAA5HKQDL4 . You are receiving this because you authored the thread.Message ID: <constructorfleet/HomeAssistant-Component-OpenAIOverride/issues/1/1736776053 @github.com>

[ltjessem]

json.decoder.JSONDecodeError: Expecting property name enclosed in double
quotes

Well the error is fairly obvious, tracking it down might be trickier though. Could you share the relevant debugging output? In mine the JSON that OpenAI returns formatted with escaped double quotes: \"service\": \"light.turn_off\" The examples I provided in my prompt in the HA GUI have unescaped double quotes: "service": "light.turn_off"

[tanker11]

Indeed, it is obvious, but I am not yet sure if it should be formatted within the code or by the command in the prompt. The latter seems to be less solid.

Here is the relevant part of the log:

{
  "finish_reason": "stop",
  "index": 0,
  "message": {
    "content": "{\n  \"service\": \"cover.close_cover\",\n  \"data\":

{\n \"entity_id\": \"cover.1_dolgozo_ikarus\"\n }\n}", "role": "assistant" } }

Not only escaped double quotes but new line chars as well. How about using one of the suggested formatting here: https://stackoverflow.com/questions/39491420/python-jsonexpecting-property-name-enclosed-in-double-quotes

Maybe an improved one to eliminate new lines, too.

ltjessem @.***> ezt írta (időpont: 2023. szept. 28., Cs, 10:14):

json.decoder.JSONDecodeError: Expecting property name enclosed in double quotes

Well the error is fairly obvious, tracking it down might be trickier though. Could you share the relevant debugging output? In mine the JSON that OpenAI returns formatted with escaped double quotes: \"service\": \"light.turn_off\" The examples I provided in my prompt in the HA GUI have unescaped double quotes: "service": "light.turn_off"

— Reply to this email directly, view it on GitHub https://github.com/constructorfleet/HomeAssistant-Component-OpenAIOverride/issues/1#issuecomment-1738685405, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVM6O36EMIJ553EY5DJIM3X4UWXJANCNFSM6AAAAAA5HKQDL4 . You are receiving this because you authored the thread.Message ID: <constructorfleet/HomeAssistant-Component-OpenAIOverride/issues/1/1738685405 @github.com>

[ltjessem]

Yeah those newlines will for sure cause issues. Not sure how or why those are turning up, are the examples in your prompt in a single line without newlines? Like the examples from the website:

Example:
Oh sure, controlling the living room tv is what I was made for.
{"service": "media_player.pause", "entity_id": "media_player.living_room_tv"}

Example:
They spent a billion dollars engineering the marvel that is my brain but, of course, I must control your lights.
{"service": "light.turn_off", "entity_id": "light.kitchen_light_homekit"}

You can also straight up just mitigate the newlines like this around line 52 in init.py: This probably isn't such a bad idea as there shouldn't be newlines there regardless.

            if segment.startswith("{"):
                json_segment = segment.replace("\n", "") # Removes newlines within the JSON segment
                service_call = json.loads(json_segment) # Change from segment to our "sanitized" json_segment

[tanker11]

I confirm there are no newlines in my generated prompt.

And the more I test the crazier the answers, an example: {"plain": {"speech": "Rendben, leengedem a dolgoz\u00f3 red\u0151nyt.\n\n{\n \"service\": \"cover.set_cover_position\",\n \"data\": {\n \"entity_id\": \"cover.1_dolgozo_ikarus\",\n \"position\": 0\n }\n}", "extra_data": null}}

This means the sanitization should be made earlier than the segment is created as this blocks the correct segmentation. Could you help achieving a correct segmentation as I am not very good in manipulating the result.response.speech["plain"]["speech"]

ltjessem @.***> ezt írta (időpont: 2023. szept. 28., Cs, 11:35):

Yeah those newlines will for sure cause issues. Not sure how or why those are turning up, are the examples in your prompt in a single line without newlines? Like the examples from the website:

Example: Oh sure, controlling the living room tv is what I was made for. {"service": "media_player.pause", "entity_id": "media_player.living_room_tv"}

Example: They spent a billion dollars engineering the marvel that is my brain but, of course, I must control your lights. {"service": "light.turn_off", "entity_id": "light.kitchen_light_homekit"}

You can also straight up just mitigate the newlines like this around line 52 in init.py: This probably isn't such a bad idea as there shouldn't be newlines there regardless.

        if segment.startswith("{"):
            json_segment = segment.replace("\n", "") # Removes newlines within the JSON segment
            service_call = json.loads(json_segment) # Change from segment to our "sanitized" json_segment

— Reply to this email directly, view it on GitHub https://github.com/constructorfleet/HomeAssistant-Component-OpenAIOverride/issues/1#issuecomment-1738811302, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVM6OZ57KCNLA2HO5HG3QDX4VAFPANCNFSM6AAAAAA5HKQDL4 . You are receiving this because you authored the thread.Message ID: <constructorfleet/HomeAssistant-Component-OpenAIOverride/issues/1/1738811302 @github.com>

[tanker11]

Played around a bit, and this is where I am so far:

just before the for loop:

    content = ""
    segments = result.response.speech["plain"]["speech"].replace("\n",

"").replace("{", "\n{", 1).replace('\"', '"').splitlines()

This is to remove the newlines and replace the escaped double quotes with simple double quotes. The trick is to replace the first { with \n{ so the splitlines work well.

But this is not enough, I still needed to further sanitize:

        if segment.startswith("{"):
            json_segment = segment.replace(' ', '').replace('`','') #

Replaces extra spaces and ending '`' characters service_call = json.loads(json_segment) # Change from segment to our "sanitized" json_segment

..as extra spaces and 3x ending ` were still there.

But I guess this is not the nicest code that can do the necessary sanitization... :/

ltjessem @.***> ezt írta (időpont: 2023. szept. 28., Cs, 11:35):

Yeah those newlines will for sure cause issues. Not sure how or why those are turning up, are the examples in your prompt in a single line without newlines? Like the examples from the website:

Example: Oh sure, controlling the living room tv is what I was made for. {"service": "media_player.pause", "entity_id": "media_player.living_room_tv"}

Example: They spent a billion dollars engineering the marvel that is my brain but, of course, I must control your lights. {"service": "light.turn_off", "entity_id": "light.kitchen_light_homekit"}

You can also straight up just mitigate the newlines like this around line 52 in init.py: This probably isn't such a bad idea as there shouldn't be newlines there regardless.

        if segment.startswith("{"):
            json_segment = segment.replace("\n", "") # Removes newlines within the JSON segment
            service_call = json.loads(json_segment) # Change from segment to our "sanitized" json_segment

— Reply to this email directly, view it on GitHub https://github.com/constructorfleet/HomeAssistant-Component-OpenAIOverride/issues/1#issuecomment-1738811302, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVM6OZ57KCNLA2HO5HG3QDX4VAFPANCNFSM6AAAAAA5HKQDL4 . You are receiving this because you authored the thread.Message ID: <constructorfleet/HomeAssistant-Component-OpenAIOverride/issues/1/1738811302 @github.com>

[tanker11]

My last mail was not quite right. What I described there ruins the answer because starts removing chars even if no control intent.

So the best would be to split the response string into [something] {control} [remaining something]. In other words it would be good to recognise the first opening { and the last closing } and everything between should go into one line and try to call the service. The rest should remain intact. Do you agree?

ltjessem @.***> ezt írta (időpont: 2023. szept. 28., Cs, 11:35):

Yeah those newlines will for sure cause issues. Not sure how or why those are turning up, are the examples in your prompt in a single line without newlines? Like the examples from the website:

Example: Oh sure, controlling the living room tv is what I was made for. {"service": "media_player.pause", "entity_id": "media_player.living_room_tv"}

Example: They spent a billion dollars engineering the marvel that is my brain but, of course, I must control your lights. {"service": "light.turn_off", "entity_id": "light.kitchen_light_homekit"}

You can also straight up just mitigate the newlines like this around line 52 in init.py: This probably isn't such a bad idea as there shouldn't be newlines there regardless.

        if segment.startswith("{"):
            json_segment = segment.replace("\n", "") # Removes newlines within the JSON segment
            service_call = json.loads(json_segment) # Change from segment to our "sanitized" json_segment

— Reply to this email directly, view it on GitHub https://github.com/constructorfleet/HomeAssistant-Component-OpenAIOverride/issues/1#issuecomment-1738811302, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVM6OZ57KCNLA2HO5HG3QDX4VAFPANCNFSM6AAAAAA5HKQDL4 . You are receiving this because you authored the thread.Message ID: <constructorfleet/HomeAssistant-Component-OpenAIOverride/issues/1/1738811302 @github.com>

[ltjessem]

Some of the craziness could be the use of non-latin characters, but those are not in the response, so that may not be an issue. I think we need to be working from the prompt side rather than trying to modify the output too much. Looking back at the responses you're getting I think that's where we need to start, as your JSON doesn't look like mine, even without the extra escaped characters.

Can you revert back to this __init__.py, and copy/paste this prompt exactly as follows:

{%- set exposed_entities = [
    "cover.1_dolgozo_ikarus"
] -%}

You are Bob, a home assistant AI.

If the user's intent is to control the home and Bob is not asking for more information, the following absolutely must be met:
* Bobs response should also acknowledge the intention of the user. If unsure, he will ask for confirmation.
* Append the user's command as Home Assistant call_service JSON structure to the response.

Here's the status of the home and the only device Bob can control.
{%- for entity_id in exposed_entities %}
{{ state_attr(entity_id, "friendly_name") }} ({{ entity_id }}) = {{ states(entity_id) }}
{%- endfor %}

Start examples:
User: Close the cover
Bob: No problem, closing the cover.
{"service": "cover.close_cover", "entity_id": "cover.1_dolgozo_ikarus"}

User: Open the cover
Bob: Sure, opening the cover.
{"service": "cover.open_cover", "entity_id": "cover.1_dolgozo_ikarus"}

User: Turn on the light
Bob: I can't do that.

End examples.

Setting the language to English may of may not be needed for this test, but you should get a debugging output similar to this (and the action should happen):

"content": "Sure, opening the cover.\n{\"service\": \"cover.open_cover\", \"entity_id\": \"cover.1_dolgozo_ikarus\"}"

[tanker11]

I also realized that OpenAI tried to provide a structured answer with indents, etc. This caused the problem. But trying just as you suggested worked like a charm, so far without any glitch!

You are an OpenAI prompt magician!

Thanks. I will play with it further and if anything useful, I will let you know.

ltjessem @.***> ezt írta (időpont: 2023. szept. 29., P, 8:00):

Some of the craziness could be the use of non-latin characters, but those are not in the response, so that may not be an issue. I think we need to be working from the prompt side rather than trying to modify the output too much. Looking back at the responses you're getting I think that's where we need to start, as your JSON doesn't look like mine, even without the extra escaped characters.

Can you revert back to this init.py https://github.com/ltjessem/HomeAssistant-Component-OpenAIOverride/blob/patch-1/__init__.py, and copy/paste this prompt exactly as follows:

{%- set exposed_entities = [ "cover.1_dolgozo_ikarus" ] -%}

You are Bob, a home assistant AI.

If the user's intent is to control the home and Bob is not asking for more information, the following absolutely must be met:

• Bobs response should also acknowledge the intention of the user. If unsure, he will ask for confirmation.
• Append the user's command as Home Assistant call_service JSON structure to the response.

Here's the status of the home and the only device Bob can control. {%- for entity_id in exposed_entities %} {{ state_attr(entity_id, "friendly_name") }} ({{ entity_id }}) = {{ states(entity_id) }} {%- endfor %}

Start examples: User: Close the cover Bob: No problem, closing the cover. {"service": "cover.close_cover", "entity_id": "cover.1_dolgozo_ikarus"}

User: Open the cover Bob: Sure, opening the cover. {"service": "cover.open_cover", "entity_id": "cover.1_dolgozo_ikarus"}

User: Turn on the light Bob: I can't do that.

End examples.

Setting the language to English may of may not be needed for this test, but you should get a debugging output similar to this (and the action should happen):

"content": "Sure, opening the cover.\n{\"service\": \"cover.open_cover\", \"entity_id\": \"cover.1_dolgozo_ikarus\"}"

— Reply to this email directly, view it on GitHub https://github.com/constructorfleet/HomeAssistant-Component-OpenAIOverride/issues/1#issuecomment-1740353394, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVM6O7TTZAWCOKUD4KCULTX4ZPYHANCNFSM6AAAAAA5HKQDL4 . You are receiving this because you authored the thread.Message ID: <constructorfleet/HomeAssistant-Component-OpenAIOverride/issues/1/1740353394 @github.com>

[Teagan42]

Sorry I didn’t see this issue get opened! I will review comments and see to it! Thanks

[Teagan42]

@ltjessem Looks like this #2 did the trick - going to close this issue.