Google Generative AI returns "400 bad request" on system control prompt

[jhbruhn]

The problem

When trying to use the Assist control functionality with the predefined prompt and enabling the Google Generative AI Assist Agent to control the HA System, it always responds with "Sorry, I had a problem talking to Google Generative AI: 400 Request contains an invalid argument.".

When I use the Generative AI without the default HA system prompt (which allows controllability of the HA instance), it works fine.

I tried the newest beta and the latest stable release of 2024.6. I tried both the recommended model settings and custom settings with content blocking set to "none".

What version of Home Assistant Core has the issue?

2024.7.0b2

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

google_generative_ai

Link to integration documentation on our website

No response

Diagnostics information

I downloaded the log (from 2024.6, 2024.7 does not log the system prompt anymore) to hopefully aid in debugging.

generative_ai.log

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

[home-assistant[bot]]

Hey there @tronikos, mind taking a look at this issue as it has been labeled with an integration (google_generative_ai_conversation) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `google_generative_ai_conversation` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign google_generative_ai_conversation` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

google_generative_ai_conversation documentation google_generative_ai_conversation source _{^{(message by IssueLinks)}}

[craig0r]

I've also recently started receiving this issue. Testing the API via google's sample command works fine:

curl -H 'Content-Type: application/json' -d '{"contents":[{"parts":[{"text":"Explain how AI works"}]}]}' -X POST 'https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-flash-latest:generateContent?key=<MY_API_REDACTED>'

I've removed and re-added the integration with my (tested and working) API key, but no dice.

[saschaabraham]

Same here, the cause is the value/name of an entity. I have removed almost all entities from the assistant, and now it works again and is able to anser simple question (time).

So next step find the entity causing the error...

[knoop7]

Same here, the cause is the value/name of an entity. I have removed almost all entities from the assistant, and now it works again and is able to anser simple question (time).

So next step find the entity causing the error...

Thank you, but it cannot be controlled after deletion. Can you? I tried and found indifferent. I would like to share more, but there is basically little research on the model.

[saschaabraham]

If it not in the list of available entities, it can not be cobtrolled I have readded my entities, but i have missed the faulty one, because it is working in my case now. But i legt out the humidity entities.

[jhbruhn]

I tried to remove the humidity sensors but it doesn't make a difference in my case. I will have to do more testing by manually adding entities later. My guess is that there is some encoding/escaping error somewhere, but unfortunately the Google API Reference is not very useful in that regard.

[tronikos]

It's caused by https://github.com/home-assistant/core/pull/118936 by @Shulyaka If I comment out tools.append(ScriptTool(self.hass, state.entity_id)) in llm.py it works. Until this is fixed the mitigation is to unexpose your scripts

[saschaabraham]

It's caused by #118936 by @Shulyaka If I comment out tools.append(ScriptTool(self.hass, state.entity_id)) in llm.py it works. Until this is fixed the mitigation is to unexpose your scripts

Yes, this is the cause, i didn't expose the scripts to the assistant while i was searching for the cause...

[Shulyaka]

Hi! Do you have scripts with variables? Which variables do you have?

[tronikos]

I just realized that Gemini doesn't allow anymore function calling without args. This means 2024.6 is also broken if you use intent_script which don't have args. Luckily not many users have intent_script. The beta made it worse because most users have regular scripts. That's why people just noticed it in beta.

[jhbruhn]

Ah yes, thanks. I had to disable my intent_script and the other scripts, all without parameters. Now it is working again with this workaround.

[tronikos]

I can confirm in beta that scripts with fields work fine. It's scripts without fields that cause this 400 error on Gemini.

[tronikos]

Can someone please confirm whether this is fixed in the latest beta?

[jhbruhn]

I have reenabled my scripts but the issue still persists with 2024.7b5 :(

Additional curiosity: my Wyoming satellite always answers with the HA Command function call instead of the answer text. Not sure if this is related or I should open another issue. It works fine (aside error 400 with scripts) for assist on the HA Frontend.

[tronikos]

Anyone else? For me the fix worked.

[jhbruhn]

Trying to boil down which scripts show the issue, I could boil it down to scripts in this format:

'1702986474849':
  alias: L-Thyroxin neue Packung
  sequence:
  - service: counter.set_value
    target:
      entity_id: counter.l_thyroxin
    data:
      value: '{{states(''counter.l_thyroxin'')|int + states(''input_number.l_thyroxin_packungsgrosse'')|int}}'
  mode: single
  icon: mdi:new-box

If I add a letter in front of the scripts name, the API does not return error 400 anymore. So it must be related to functions having a name which does not start with a letter. The workaround is easy: I just change my script's ids (they're bad anyways :D). But IIRC these were automatically generated by HA, so it might make sense to either prepend a letter in the LLM API, or change how HA generates default script ids.

[LittleDonnieJ]

I updated mine to the beta a couple hours ago and tested and it's still returning a 400 error. I drilled in to see where it may be breaking by disabling my 3 scripts I had exposed. That allowed it to work. I then added back in one by one to see which broke it and found it. It seems that it's breaks when presented with an entity ID with leading or all digits for scripts. I further tested by renaming my script with all digits to alpha, and underscores, and that worked. I also took one of the working named scripts and added a digit at the front of the EntityID and that broke it again. Hope this helps!