Streaming: Unable to connect to homeassistant:8123

[helicalchris]

Streaming: Unable to connect to homeassistant:8123

Bug Description

I have just upgraded to 1.2 to use the new streaming feature. When I run an automation with the streaming feature I get an error in my logs:

Logger: homeassistant.components.automation.gemini_v2
Source: components/automation/__init__.py:763
integration: Automation ([documentation](https://www.home-assistant.io/integrations/automation), [issues](https://github.com/home-assistant/core/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+automation%22))
First occurred: 15:47:22 (4 occurrences)
Last logged: 16:26:53

Error while executing automation automation.gemini_v2: Fetch failed: Cannot connect to host homeassistant.local:8123 ssl:default [Invalid argument]

Version: 1.2

Service Call

action: llmvision.stream_analyzer
data:
  include_filename: false
  target_width: 1280
  detail: high
  max_tokens: 100
  temperature: 0.2
  provider: 01JAG0V6JZTY81JBD541P77R1M
  message: >-
    Motion has been detected on my front camera. 5 of the following pictures
    are  named family members and the video is from the front of the house where
    movement occurred. Very briefly describe what you see. If a person or animal
    is present, describe them in detail. If a person's clothing or a vehicle has
    a logo or text then describe it. Feel free to name any family members that
    may be present.  Do not describe stationary objects or buildings except if
    they have a logo. Humour and sarcasm is encouraged unless something sinister
    is occurring. Your message needs to be short enough to fit in a phone
    notification approx 100 characters.
  image_entity:
    - camera.front_camera_rt
  interval: 3
  duration: 3
response_variable: response

Logs

Logger: homeassistant.components.automation.gemini_v2
Source: components/automation/__init__.py:763
integration: Automation ([documentation](https://www.home-assistant.io/integrations/automation), [issues](https://github.com/home-assistant/core/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+automation%22))
First occurred: 15:47:22 (4 occurrences)
Last logged: 16:26:53

Error while executing automation automation.gemini_v2: Fetch failed: Cannot connect to host homeassistant.local:8123 ssl:default [Invalid argument]

Additional context

homeassistant:8123 should actually work and while I don't know why there is a call to it at all, I am also confused why said call doesn't work. The camera referenced is a ring camera via ring_mqtt to make an rtsp stream, and when I go to the entity directly in HA the stream works, so I can't see why it would affect llm_vision. However ring_mqtt can have a short delay between requesting video and showing it due to lag and buffering - could that be a cause?

_I am also having different issues with running the old video 'from file' methods after upgrade. I'm not seeking to fix here, and the error are different, but in case it becomes relevant (maybe because I have a deeper issue post upgrade), I get the error UnboundLocalError: cannot access local variable 'responsetext' where it is not associated with a value

[valentinfrlch]

Thanks for the logs! The call happens because llmvision needs to access the frames from the camera. It does that by calling the home assistant api and requesting the 'entity_picture' attribute of the camera entity. Can you check in your Home Assistant Settings > Network what you have for Local Network? This is the base_url llmvision will try to use for the api. In some cases setting this to automatic wouldn't work. I have set it to the (static) IP address.

As for the second issue: Have you set up your providers again? Unfortunately, the old configurations are incompatible with v1.2 so you might need to set some of them up again.

[helicalchris]

Ahhh we may have a problem between how you are getting frames and how ring_mqtt operates. It does it through the add on by hosting an rtsp stream, and not as part of the offical Ring integration entities in HA. You actually create a new Generic Camera entity and in that configure the URIs for snapshots and for live streaming. I don't know but I assume that this is different to how an NVR like frigate is integrating cameras into HA?

(I have done the reconfiguration for the update, I actually deleted the old device and created a new one for Gemini just in case. Edit: just realised that because I remade the device the provider changed and so the automation using it needed to point at the new name. Muppet. Fixed!)

[helicalchris]

My configured URI for generic camera live stream is rtsp://03cabcc9-ring-mqtt:8554/54e0190bc5c4_live

I could make this externally accessible on the same port on homeassistanr.local but that's probably not the problem here

[valentinfrlch]

To check if it should work with your camera you can go to the developer tools and search for you camera entity. It should look roughly like this:

Most importantly the camera entity should have the entity_picture attribute. Without it, it won't work.

The uri does not need to be publicly accessible and it's probably better if it isn't.

[helicalchris]

Yeah so Generic Cameras don't have that. Hence why there is an issue here.

Well at least we know this doesn't support Ring cameras now 😀

From the device diagnostic (and before anyone asks, the reason for the still image path not being right is that it's a placeholder while I work out why I find snapshots make my battery drain on the camera!)

data": { "title": "generic_front_RT", "data": {}, "options": { "use_wallclock_as_timestamps": false, "still_image_url": "https://freedesignfile.com/", "stream_source": "rtsp://03cabcc9-ring-mqtt:8554/", "rtsp_transport": "tcp", "authentication": "basic", "framerate": 20.0, "verify_ssl": false, "limit_refetch_to_url_change": false, "content_type": "image/png" } } }

[valentinfrlch]

Huh weird. So if I understand correctly you have an rtsp stream for your cameras? Can you use 'Generic Camera' to create a camera entity from that stream? I just did that to test and it works for me: (entity_picture is present)

[helicalchris]

Oh that's interesting. I assumed that a Generic Camera just didn't offer it - that is what I use now to make an entity to host the rtsp stream in HA as an object.

I honestly don't know...I'll go off and research and when I created mine I didn't get that. I wonder why when you did. I just followed the ring_mqtt install instructions but maybe if I did something else it would appear

[helicalchris]

Oh hang on, turns out under developer tools there is an entity_picture, under diagnostics there isn't. I had diagnostics open, so used that. Serves me right.

access_token: redacted
frontend_stream_type: hls
entity_picture: >-
  /api/camera_proxy/camera.front_camera_rt?token=redacted
friendly_name: Front Camera RT
supported_features: 2

If I browse to that entity_picture path from a browser I get 403 forbidden. Should that happen - I don't have another camera type to test against

[valentinfrlch]

When you browse to that path it should show the image. I had that same issue as well at some point and iirc a restart helped. The 403 status code is 'forbidden' so maybe the token expired.

[helicalchris]

Ok yes I think you were right. Tried again this morning and the entity_picture is working. Token must have expired.

So still left wondering why I'm getting the error I raised here

I'll do a bit more testing but open to ideas.

[helicalchris]

A thought. Are you going to https:// or http? I run nginx so internally homeassistant.local and IP are http on 8123 and the proper url is https on 443 (for access externally)

[valentinfrlch]

This depends on what you have in your Network Settings: This will be the base url for all requests to the Home Assistant API.

You can see the full call by enabling logging:

logger:
  logs:
    custom_components.llmvision: debug

[helicalchris]

Got it - ye that was the issue. I've never had a problem before so it's obviously not heavily used by people but it was set to https.

Thanks, that would have taken me forever to work out!!

Chris