Image error failure for video files

[helicalchris]

Bug Description

When I run a video action call to LLM Vision I get an error in the traces (and logs):

Stopped because an error was encountered at 29 October 2024 at 19:35:44 (runtime: 11.03 seconds)

No image input provided

I write my video file to /share/front-camera-clip.mp4 (and I can load that file and play it myself and it works), and it was working fine until recently. It makes no difference if I use /www/llmvision

It works in image mode if I point it at an image in /share

action: llmvision.image_analyzer
data:
  provider: (redacted not sure if this is secret or a uid!)
  message: tell me what you see
  image_file: /share/front-snapshot.jpg
  include_filename: false
  max_tokens: 86
  temperature: 0.4

It fails in video mode if I point it at a video in /share

action: llmvision.video_analyzer
data:
  max_frames: 3
  include_filename: false
  target_width: 1280
  detail: low
  max_tokens: 100
  temperature: 0.2
  provider: (redacted not sure if this is secret or a uid!)
  message: tell me what you see
  video_file: /share/front-camera-clip.mp4

I have also seen the following errors shown underneath - note different folder names tmp_clips and tmp_frames. Is this relevant?

The automation trace is still giving the same error as before but is there an issue with inconsistent folder names?

Version: 1.3 (works in 1.1.x)

Logs

2024-11-04 15:59:15.979 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Processing videos: ['/config/www/llmvision/front-camera-clip.mp4']
2024-11-04 15:59:15.980 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Created /config/custom_components/llmvision/tmp_frames
2024-11-04 15:59:16.293 DEBUG (MainThread) [custom_components.llmvision.media_handlers] Adding frame frame0001.jpg
2024-11-04 15:59:16.307 INFO (MainThread) [custom_components.llmvision.media_handlers] Failed to delete tmp folder: [Errno 2] No such file or directory: '/config/custom_components/llmvision/tmp_clips'
2024-11-04 15:59:16.308 INFO (MainThread) [custom_components.llmvision.media_handlers] Deleted tmp folder: /config/custom_components/llmvision/tmp_frames
2024-11-04 15:59:16.309 INFO (MainThread) [custom_components.llmvision.request_handlers] Provider from call: Google

Additional context

Discussions for this started in my feature request 75 here

[valentinfrlch]

Thanks! tmp_clips and tmp_frames are both intentionally different folders. Clips will temporarily store any clips fetched from Frigate, whereas frames will store the extracted frames from clips (including videos from video_path).

You can ignore the 'Failed to delete' message as the folder was not created and therefore not deleted. (I should still fix this though...) Sorry you have to deal with this, I hope it is not a widespread issue...

I'll try to recreate your exact circumstances tomorrow. Thanks for all the logs you submitted, it helps a lot!

[helicalchris]

I'll also try to look at your code and see if I can't pin down which line is causing the error. I assume it can't be too hard as the file is probably never opened so it must be early in the logic. Just can't do it right now on my phone 😀

[valentinfrlch]

Thank you! The relevant code is in the media_handlers.py file and the function is mostly add_videos.

[helicalchris]

The error is being generated from request_handlers.py in the _validate_call function because base64_images is empty

        if base64_images == []:
            raise ServiceValidationError(ERROR_NO_IMAGE_INPUT)

If I add

        _LOGGER.debug(f"Check images {base64_images}")

before it then the logs show:

2024-11-05 15:50:56.930 DEBUG (MainThread) [custom_components.llmvision.request_handlers] Check images []

Now I just need to work backwards to why it's empty. But it proves its nothing to do with the clip being loaded and other debug logs I have done shows everything works fine until the point just before the call is made to Google when the validate happens and the empty array is discovered

[valentinfrlch]

Thanks for looking into this! I have identified (and hopefully fixed) some I/O operations that where not executed in home assistant's loop. Some of them were in the function that decides how similar two images are. I have pushed the modified file already. Could you test if that solves the issue? I don't want to create a beta just yet as there are some other things I want to work on but here is the raw file: https://raw.githubusercontent.com/valentinfrlch/ha-llmvision/refs/heads/data-analyzer/custom_components/llmvision/media_handlers.py

You can paste this into the media_handlers.py file.

[helicalchris]

Will do. The sorted_frames array is empty so lets see if this helps..will try now

[helicalchris]

No difference. Is it possibly because there is only a little difference in the footage - it's only a short clip I use and in the latest one the car door opens in the gloomy darkness. Is there logic that there is always at least a minimum of at least frame saved even if all frames seem identical?

Edit: I just changed clip length from 3 sec to 10 and it has started working so there may be some logic to the above being true

[valentinfrlch]

Ahh well that is another possibility. If the clip is too short it wont extract enough files. Guess it's simpler than I imaged. I will adjust the interval so that more images are extracted for short clips.

[helicalchris]

I think it's not maybe related to length it's related to activity (of which there will be less in a short clip typically, to be fair). Can you not put in more simply that if there are 0 frames in the array then add (any) one frame in there?

Reason I say that is given the lag for ring to start sometimes clips could be exceptionally short (theoretically at least, I haven't seen that) so just beware of that I guess.

[valentinfrlch]

Did excactly that. Can you try again with the updated media_handler.py? You can find the raw file here: https://raw.githubusercontent.com/valentinfrlch/ha-llmvision/refs/heads/data-analyzer/custom_components/llmvision/media_handlers.py

[helicalchris]

Sorry I haven't forgotten, I did try yesterday, but using my phone to copy and replace the whole py file,and it was the same .... but I don't trust it saved properly so I am going to repeat using a laptop when I get a chance to be sure before I report back!