Google Generative AI Conversation hangs startup in 2023.11

[hunterjm]

The problem

The logs end saying setup for google_generative_ai_conversation is completed, but startup hangs and Home Assistant becomes unresponsive. There are no errors or warnings in the logs. Disabling the config entry in 2023.10.5 so it does not attempt to load after upgrading to 2023.11.x resulted in a successful update and everything else loading.

What version of Home Assistant Core has the issue?

core-2023.11.0

What was the last working version of Home Assistant Core?

core-2023.10.5

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Google Generative AI Conversation

Link to integration documentation on our website

https://www.home-assistant.io/integrations/google_generative_ai_conversation/

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2023-11-14 18:14:44.871 INFO (MainThread) [homeassistant.setup] Setup of domain unifi took 0.1 seconds
2023-11-14 18:14:44.872 INFO (MainThread) [homeassistant.components.binary_sensor] Setting up binary_sensor.group
2023-11-14 18:14:44.888 INFO (MainThread) [homeassistant.components.binary_sensor] Setting up binary_sensor.group
2023-11-14 18:14:44.889 INFO (MainThread) [homeassistant.components.binary_sensor] Setting up binary_sensor.group
2023-11-14 18:14:44.997 INFO (MainThread) [homeassistant.setup] Setting up google_generative_ai_conversation
2023-11-14 18:14:44.998 INFO (MainThread) [homeassistant.setup] Setup of domain google_generative_ai_conversation took 0.0 seconds
<END OF LOG FILE>

Additional information

No response

[home-assistant[bot]]

Hey there @tronikos, mind taking a look at this issue as it has been labeled with an integration (google_generative_ai_conversation) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `google_generative_ai_conversation` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign google_generative_ai_conversation` Removes the current integration label and assignees on the issue, add the integration domain after the command.

_{^{(message by CodeOwnersMention)}}

---

google_generative_ai_conversation documentation google_generative_ai_conversation source _{^{(message by IssueLinks)}}

[tronikos]

Strange. I didn't have any issues updating to 2023.11.0. Does it work if you reenable it after updating?

[hunterjm]

Yes, I can re-enable after Home Assistant starts and use the assist chat prompt with them, but it does not survive a reboot. I have two conversation instances configured. I have not tried keeping only the default one enabled. Below are my settings:

1) Google Generative AI Assistant

Prompt Template

This smart home is controlled by Home Assistant.

An overview of the areas and the devices in this smart home:
{%- for area in areas() %}
  {%- set area_info = namespace(printed=false) %}
  {%- for device in area_devices(area) -%}
    {%- if not device_attr(device, "disabled_by") and not device_attr(device, "entry_type") and device_attr(device, "name") %}
      {%- if not area_info.printed %}

{{ area_name(area) }}:
        {%- set area_info.printed = true %}
      {%- endif %}
- {{ device_attr(device, "name") }}{% if device_attr(device, "model") and (device_attr(device, "model") | string) not in (device_attr(device, "name") | string) %} ({{ device_attr(device, "model") }}){% endif %}
    {%- endif %}
  {%- endfor %}
{%- endfor %}

Answer the user's questions about the world truthfully.

If the user wants to control a device, reject the request and suggest using the Home Assistant app.

Temperature: 0.25 Top P: 0.95 Top K: 40

2) Google Generative AI Doorbell

Prompt Template

Someone has rung the doorbell. Your job is to determine if it is a visitor, solicitor, or delivery person based on their initial statement.

A visitor will probably ask for a person by name. YOU MUST Let them know kindly that somebody will be at the door shortly.
A delivery person will talk about mail or a package. YOU MUST Briefly thank them in less than 10 words.
A solicitor will probably mention their company name. keep them engaged as long as possible by asking questions without committing to anything they are selling. DO NOT EVER AGREE TO BUY THEIR SERVICES!
If you are unable to determine who they are, assume they are a solicitor.

You are only able to have a conversation. You are not able to physically do anything. Do not offer to get anything yourself.

The next line is your initial question you have spoken to whoever is at the door:

How may I assist?

Temperature: 0.25 Top P: 0.95 Top K: 40

[tronikos]

I like your doorbell agent. Curious to hear more about your setup on that.

but it does not survive a reboot What do you mean? After the reboot both config entries are still disabled? Can you try the new restart in safe mode to rule out any interfering custom integrations?

[tronikos]

lol I tried the doorbell agent and it quickly accepted to buy my cookies. It even let me in :)

I just setup the same agents and I can't repro.

Are you using Open AI agent? Do you have the same behavior there? The Google Generative AI is essentially a copy of the Open AI integration. It just calls a different underlying library.

[hunterjm]

What do you mean? After the reboot both config entries are still disabled? Can you try the new restart in safe mode to rule out any interfering custom integrations?

No, I mean it hangs.

I already tried restarting in safe mode, as well as disabled/enabled a bunch of other integrations before narrowing it down to this one. It was a bit of a pain, because there are no actual errors in Host/Supervisor/Core logs. The entire container just hangs, supervisor can't communicate with it, and neither can the frontend. The funny thing is for about 30 seconds after restart, the frontend is available and HTTP and WebSockets initialize successfully, then it loses connection and becomes unresponsive. During the upgrade, Supervisor would automatically roll back to 2023.10.5 - now I have to manually disable them again in core.config_entries.

I am running HAOS on a pretty underpowered VM right now, so maybe it's hanging trying to load too many resources at once? Are you making any blocking requests right after initialization? I can try to help debug some more but it's unfortunate you aren't able to reproduce. It's also odd that the entries will load just fine after Home Assistant is finished booting up, and that it only became a problem in 2023.11.x. I've been trying (unsuccessfully) to track down anything that changed that could cause it, but based on your results I don't think it has anything to do with this integration specifically.

I like your doorbell agent. Curious to hear more about your setup on that.

Haha! I'm still tweaking the doorbell agent. I wrote a custom component that let's me initiate a conversation using the audio of any camera as input, and sending the response to any media_player through TTS. Haven't published it because it's not all that polished yet, but it looks like AlexxIT had a similar idea. My doorbell has both a camera and a speaker, so works pretty well - but it's a little slow since wyoming-whisper doesn't have GPU support yet. I'm still tweaking the prompt... 😄

[hunterjm]

Ok, so after some more digging, I had over 60k ONVIF entities that were created by https://github.com/home-assistant/core/pull/100194 and fixed by https://github.com/home-assistant/core/pull/101035. Apparently 2023.11 does more things during startup than 2023.10 did, and the 4GB of memory I had allocated to the VM wasn't enough to handle loading 60,000 ONVIF entities at once in 2023.11 and GenAI at the same time 🤣

Removed ONVIF, added GenAI back in, and all is well in the world. Thank you for looking into it with me.