search

valentinfrlch / ha-llmvision

Let Home Assistant see!
Apache License 2.0
161 stars 6 forks source link

Failed to call service llmvision.video_analyzer. None #43

Closed darkneo29 closed 3 months ago

darkneo29 commented 3 months ago

Describe the bug When calling the service LLM Vision: Video Analyzer and using Google as the provider I get this error Failed to call service llmvision.video_analyzer. None. When I use OPENAI it works

Version 1.0.1

Logs


Paste relevant logs here
Maybe this??
Logger: homeassistant.helpers.script.websocket_api_script
Source: helpers/script.py:2002
First occurred: 9:38:18 AM (9 occurrences)
Last logged: 9:55:04 AM

websocket_api script: Error executing script. Error for call_service at pos 1: None
valentinfrlch commented 3 months ago

What did you use as model? You can leave the parameter blank to use the default model for the selected provider. There is a known bug right now that will throw this error instead of properly stating that the model doesn't exist.

Let me know if that works.

darkneo29 commented 3 months ago

Hey, I am using gemini-1.5-flash and Pro and error on both but when I try via image with an image it works with this models.. I tried blank as well and no luck same error

valentinfrlch commented 3 months ago

Can you please provide the service call so I can reproduce the error? This works for me:

service: llmvision.image_analyzer
data:
  provider: Google
  include_filename: true
  target_width: 1280
  max_tokens: 100
  temperature: 0.5
  message: Describe the image
  image_entity:
    - camera.entity
  detail: high
darkneo29 commented 3 months ago

service: llmvision.video_analyzer data: provider: Google model: gemini-1.5-flash interval: 3 target_width: 1280 detail: high max_tokens: 300 temperature: 0.5 video_file: /config/www/snapshots/vidgarage.mp4 message: >- Review this video Clip. You are a security guard reviewing the garage camera video, your job is to keep the home safe and let us know if any suspicious activity is detected. Keep your response short and too the point. At the end of your response type either threat=detected or threat=safe include_filename: true response_variable: response

darkneo29 commented 3 months ago

The image works fine. It's the new video part that doesn't work when I select Google. When I select OpenAI, there are no issues. When I use the image function, everything works for Google as well.

valentinfrlch commented 3 months ago

I cannot reproduce the error. This works:

service: llmvision.video_analyzer
data:
  provider: Google
  model: gemini-1.5-flash
  interval: 5
  target_width: 1280
  detail: high
  max_tokens: 300
  temperature: 0.5
  video_file: /config/www/tmp/test.mp4
  message: >-
    Review this video Clip. You are a security guard reviewing the garage camera
    video, your job is to keep the home safe and let us know if any suspicious
    activity is detected. Keep your response short and too the point. At the end
    of your response type either threat=detected or threat=safe
  include_filename: true

response:

response_text: >
  The man in the video appears to be a resident of the home. He is carrying
  groceries and appears to be entering the home. 

  threat=safe

The only way I get the None error is, if I leave the model parameter blank which Home Assistant then automatically sets to null. These won't work:

...
model: null
model:
...

Either set the parameter to an existing model or leave the parameter out.

If the issue persists I need more logs as otherwise there is no way for me to understand what's going on. You can enable debug logging by adding this to your configuration.yaml:

logger:
  logs:
    custom_components.llmvision: debug
darkneo29 commented 3 months ago

Here we go

2024-08-01 04:23:33.536 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' accesses hass.helpers.event. This is deprecated and will stop working in Home Assistant 2024.11, it should be updated to import functions used from event directly at custom_components/hacs/base.py, line 630: self.hass.helpers.event.async_track_time_interval(, please create a bug report at https://github.com/hacs/integration/issues 2024-08-01 04:23:36.846 ERROR (MainThread) [homeassistant.components.google_assistant.http] Request for https://homegraph.googleapis.com/v1/devices:requestSync failed: 500 2024-08-01 04:24:12.098 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Processing videos: ['/config/www/snapshots/vidlivingroom.mp4'] 2024-08-01 04:24:12.099 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Created config/custom_components/llmvision/tmp_frames 2024-08-01 04:24:14.047 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0013.png 2024-08-01 04:24:14.330 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0006.png 2024-08-01 04:24:14.681 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0008.png 2024-08-01 04:24:15.044 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0007.png 2024-08-01 04:24:15.432 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0011.png 2024-08-01 04:24:15.760 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0009.png 2024-08-01 04:24:16.098 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0002.png 2024-08-01 04:24:16.438 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0010.png 2024-08-01 04:24:16.779 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0014.png 2024-08-01 04:24:17.069 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0004.png 2024-08-01 04:24:17.407 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0012.png 2024-08-01 04:24:17.690 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0005.png 2024-08-01 04:24:18.014 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0001.png 2024-08-01 04:24:18.341 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0003.png 2024-08-01 04:24:18.674 DEBUG (MainThread) [custom_components.llmvision.request_handlers] Base64 Images: ['', '', '', '', '', '', '', '', '', '', '', '', '', ''] 2024-08-01 04:24:18.674 INFO (MainThread) [custom_components.llmvision.request_handlers] Request data: {'contents': [{'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'The attached images are frames from a video. what do you see'}]}], 'generationConfig': {'maxOutputTokens': 164, 'temperature': 0.5}} 2024-08-01 04:24:20.331 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Error for call_service at pos 1: None 2024-08-01 04:24:20.332 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140646696627808] None 2024-08-01 04:26:22.031 ERROR (MainThread) [homeassistant.components.script] Script with alias 'morning set temperature ' failed to setup actions and has been disabled: Unknown device 'a19255cf6442d6bae6c4f36d22e546b7' 2024-08-01 04:26:25.158 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Processing videos: ['/config/www/snapshots/vidlivingroom.mp4'] 2024-08-01 04:26:25.159 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Created config/custom_components/llmvision/tmp_frames 2024-08-01 04:26:27.074 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0013.png 2024-08-01 04:26:27.350 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0006.png 2024-08-01 04:26:27.689 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0008.png 2024-08-01 04:26:28.036 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0007.png 2024-08-01 04:26:28.364 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0011.png 2024-08-01 04:26:28.690 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0009.png 2024-08-01 04:26:29.037 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0002.png 2024-08-01 04:26:29.369 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0010.png 2024-08-01 04:26:29.700 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0014.png 2024-08-01 04:26:29.994 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0004.png 2024-08-01 04:26:30.305 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0012.png 2024-08-01 04:26:30.591 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0005.png 2024-08-01 04:26:30.910 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0001.png 2024-08-01 04:26:31.235 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0003.png 2024-08-01 04:26:31.546 DEBUG (MainThread) [custom_components.llmvision.request_handlers] Base64 Images: ['', '', '', '', '', '', '', '', '', '', '', '', '', ''] 2024-08-01 04:26:31.546 INFO (MainThread) [custom_components.llmvision.request_handlers] Request data: {'contents': [{'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'The attached images are frames from a video. what do you see'}]}], 'generationConfig': {'maxOutputTokens': 164, 'temperature': 0.5}} 2024-08-01 04:26:33.038 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Error for call_service at pos 1: None 2024-08-01 04:26:33.038 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140646263265536] None

valentinfrlch commented 3 months ago

Thanks for enabling debugging. Can you try with a higher interval so that less images are sent?

You said image_analyzer works and since video_analyzer and image_analyzer share the code for requests I suspect Google might have a limit of how many images can be processed in one request.

darkneo29 commented 3 months ago

Tried at 10 and 15 and same error

2024-08-01 04:23:33.536 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' accesses hass.helpers.event. This is deprecated and will stop working in Home Assistant 2024.11, it should be updated to import functions used from event directly at custom_components/hacs/base.py, line 630: self.hass.helpers.event.async_track_time_interval(, please create a bug report at https://github.com/hacs/integration/issues 2024-08-01 04:23:36.846 ERROR (MainThread) [homeassistant.components.google_assistant.http] Request for https://homegraph.googleapis.com/v1/devices:requestSync failed: 500 2024-08-01 04:24:12.098 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Processing videos: ['/config/www/snapshots/vidlivingroom.mp4'] 2024-08-01 04:24:12.099 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Created config/custom_components/llmvision/tmp_frames 2024-08-01 04:24:14.047 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0013.png 2024-08-01 04:24:14.330 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0006.png 2024-08-01 04:24:14.681 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0008.png 2024-08-01 04:24:15.044 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0007.png 2024-08-01 04:24:15.432 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0011.png 2024-08-01 04:24:15.760 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0009.png 2024-08-01 04:24:16.098 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0002.png 2024-08-01 04:24:16.438 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0010.png 2024-08-01 04:24:16.779 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0014.png 2024-08-01 04:24:17.069 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0004.png 2024-08-01 04:24:17.407 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0012.png 2024-08-01 04:24:17.690 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0005.png 2024-08-01 04:24:18.014 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0001.png 2024-08-01 04:24:18.341 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0003.png 2024-08-01 04:24:18.674 DEBUG (MainThread) [custom_components.llmvision.request_handlers] Base64 Images: ['', '', '', '', '', '', '', '', '', '', '', '', '', ''] 2024-08-01 04:24:18.674 INFO (MainThread) [custom_components.llmvision.request_handlers] Request data: {'contents': [{'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'The attached images are frames from a video. what do you see'}]}], 'generationConfig': {'maxOutputTokens': 164, 'temperature': 0.5}} 2024-08-01 04:24:20.331 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Error for call_service at pos 1: None 2024-08-01 04:24:20.332 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140646696627808] None 2024-08-01 04:26:22.031 ERROR (MainThread) [homeassistant.components.script] Script with alias 'morning set temperature ' failed to setup actions and has been disabled: Unknown device 'a19255cf6442d6bae6c4f36d22e546b7' 2024-08-01 04:26:25.158 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Processing videos: ['/config/www/snapshots/vidlivingroom.mp4'] 2024-08-01 04:26:25.159 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Created config/custom_components/llmvision/tmp_frames 2024-08-01 04:26:27.074 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0013.png 2024-08-01 04:26:27.350 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0006.png 2024-08-01 04:26:27.689 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0008.png 2024-08-01 04:26:28.036 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0007.png 2024-08-01 04:26:28.364 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0011.png 2024-08-01 04:26:28.690 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0009.png 2024-08-01 04:26:29.037 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0002.png 2024-08-01 04:26:29.369 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0010.png 2024-08-01 04:26:29.700 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0014.png 2024-08-01 04:26:29.994 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0004.png 2024-08-01 04:26:30.305 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0012.png 2024-08-01 04:26:30.591 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0005.png 2024-08-01 04:26:30.910 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0001.png 2024-08-01 04:26:31.235 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0003.png 2024-08-01 04:26:31.546 DEBUG (MainThread) [custom_components.llmvision.request_handlers] Base64 Images: ['', '', '', '', '', '', '', '', '', '', '', '', '', ''] 2024-08-01 04:26:31.546 INFO (MainThread) [custom_components.llmvision.request_handlers] Request data: {'contents': [{'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'The attached images are frames from a video. what do you see'}]}], 'generationConfig': {'maxOutputTokens': 164, 'temperature': 0.5}} 2024-08-01 04:26:33.038 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Error for call_service at pos 1: None 2024-08-01 04:26:33.038 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140646263265536] None 2024-08-01 04:44:10.642 ERROR (MainThread) [homeassistant.components.script] Script with alias 'morning set temperature ' failed to setup actions and has been disabled: Unknown device 'a19255cf6442d6bae6c4f36d22e546b7' 2024-08-01 04:45:31.479 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Processing videos: ['/config/www/snapshots/vidgarage.mp4'] 2024-08-01 04:45:31.480 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Created config/custom_components/llmvision/tmp_frames 2024-08-01 04:45:32.035 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0013.png 2024-08-01 04:45:32.307 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0006.png 2024-08-01 04:45:32.640 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0008.png 2024-08-01 04:45:32.970 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0007.png 2024-08-01 04:45:33.309 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0011.png 2024-08-01 04:45:33.649 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0009.png 2024-08-01 04:45:33.984 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0002.png 2024-08-01 04:45:34.320 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0010.png 2024-08-01 04:45:34.664 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0014.png 2024-08-01 04:45:34.951 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0004.png 2024-08-01 04:45:35.281 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0012.png 2024-08-01 04:45:35.561 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0005.png 2024-08-01 04:45:35.887 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0001.png 2024-08-01 04:45:36.039 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0003.png 2024-08-01 04:45:36.376 DEBUG (MainThread) [custom_components.llmvision.request_handlers] Base64 Images: ['', '', '', '', '', '', '', '', '', '', '', '', '', ''] 2024-08-01 04:45:36.376 INFO (MainThread) [custom_components.llmvision.request_handlers] Request data: {'contents': [{'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'The attached images are frames from a video. what do you see'}]}], 'generationConfig': {'maxOutputTokens': 164, 'temperature': 0.5}} 2024-08-01 04:45:37.893 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Error for call_service at pos 1: None 2024-08-01 04:45:37.894 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140645861733088] None 2024-08-01 04:45:53.552 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Processing videos: ['/config/www/snapshots/vidgarage.mp4'] 2024-08-01 04:45:53.553 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Created config/custom_components/llmvision/tmp_frames 2024-08-01 04:45:54.148 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0013.png 2024-08-01 04:45:54.448 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0006.png 2024-08-01 04:45:54.787 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0008.png 2024-08-01 04:45:55.175 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0007.png 2024-08-01 04:45:55.525 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0011.png 2024-08-01 04:45:55.881 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0009.png 2024-08-01 04:45:56.223 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0002.png 2024-08-01 04:45:56.555 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0010.png 2024-08-01 04:45:56.897 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0014.png 2024-08-01 04:45:57.178 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0004.png 2024-08-01 04:45:57.502 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0012.png 2024-08-01 04:45:57.793 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0005.png 2024-08-01 04:45:58.116 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0001.png 2024-08-01 04:45:58.292 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0003.png 2024-08-01 04:45:58.612 DEBUG (MainThread) [custom_components.llmvision.request_handlers] Base64 Images: ['', '', '', '', '', '', '', '', '', '', '', '', '', ''] 2024-08-01 04:45:58.613 INFO (MainThread) [custom_components.llmvision.request_handlers] Request data: {'contents': [{'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'The attached images are frames from a video. what do you see'}]}], 'generationConfig': {'maxOutputTokens': 164, 'temperature': 0.5}} 2024-08-01 04:46:00.197 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Error for call_service at pos 1: None 2024-08-01 04:46:00.197 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140645861733088] None 2024-08-01 04:46:28.254 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Processing videos: ['/config/www/snapshots/vidlivingroom.mp4'] 2024-08-01 04:46:28.255 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Created config/custom_components/llmvision/tmp_frames 2024-08-01 04:46:29.576 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0013.png 2024-08-01 04:46:29.859 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0006.png 2024-08-01 04:46:30.242 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0008.png 2024-08-01 04:46:30.594 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0007.png 2024-08-01 04:46:30.954 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0011.png 2024-08-01 04:46:31.299 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0009.png 2024-08-01 04:46:31.646 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0002.png 2024-08-01 04:46:31.988 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0010.png 2024-08-01 04:46:32.325 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0014.png 2024-08-01 04:46:32.617 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0004.png 2024-08-01 04:46:32.952 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0012.png 2024-08-01 04:46:33.268 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0005.png 2024-08-01 04:46:33.608 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0001.png 2024-08-01 04:46:33.949 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0003.png 2024-08-01 04:46:34.276 DEBUG (MainThread) [custom_components.llmvision.request_handlers] Base64 Images: ['', '', '', '', '', '', '', '', '', '', '', '', '', ''] 2024-08-01 04:46:34.276 INFO (MainThread) [custom_components.llmvision.request_handlers] Request data: {'contents': [{'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'Video frame 0:'}, {'inline_data': {'mime_type': 'image/jpeg', 'data': ''}}]}, {'role': 'user', 'parts': [{'text': 'The attached images are frames from a video. what do you see'}]}], 'generationConfig': {'maxOutputTokens': 164, 'temperature': 0.5}} 2024-08-01 04:46:35.771 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Error for call_service at pos 1: None 2024-08-01 04:46:35.772 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140645861733088] None

darkneo29 commented 3 months ago

here is a link to the video file...maybe something is going on there?? https://drive.google.com/file/d/1pEr3A1B0JS5xpNYshiE-FSCKzeV9nL8O/view?usp=drive_link

valentinfrlch commented 3 months ago

According to the logs it is still adding more than 10 images. Try an interval of 60. If that doesn't help, downscale the images using the target_width parameter.

darkneo29 commented 3 months ago

Downscale to 1080 worked! Moving interval up or down doesn't seem to impact anything but this will do the job. Thank you. You have a buy me a coffee link??

valentinfrlch commented 3 months ago

Glad to hear it works! I’ll try to add an error message for when the input token limit is exceeded.

I don’t but you’re most welcome!