search

valentinfrlch / ha-llmvision

Let Home Assistant see!
Apache License 2.0
171 stars 6 forks source link

Gif Image Entity Input Support #54

Closed mediacutlet closed 2 months ago

mediacutlet commented 2 months ago

Bug Description

Description: Upon seeing the update available, I updated the integration and restarted Home Assistant. I tried to run the action using a gif and saw the same error as 1.0.3 as it does not support gifs. The error logs show the same or similar. It seems as though I am not actually upgraded to 1.0.4 although I have no more update prompt available in HACS. I tried completely removing the integration, deleting the repo, and reinstalling without success.

Version:
HA Version: 2024.8.2 Integration version: 1.0.4

Service Call

action: llmvision.image_analyzer
data:
  provider: OpenAI
  model: gpt-4o
  max_tokens: 100
  temperature: 0.5
  image_entity:
    - image.weather_radar_new_jersey
  message: describe this image

Logs

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/PIL/JpegImagePlugin.py", line 639, in _save
    rawmode = RAWMODE[im.mode]
              ~~~~~~~^^^^^^^^^
KeyError: 'P'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 525, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 764, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 727, in _async_run_long_action
    return await long_task
           ^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2763, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2806, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/__init__.py", line 125, in image_analyzer
    client = await processor.add_images(call.image_entities, call.image_paths, call.target_width, call.include_filename)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 106, in add_images
    base64_image=await self.resize_image(target_width=target_width, image_data=image_data),
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 69, in resize_image
    base64_image = await self._encode_image(img)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 24, in _encode_image
    img.save(img_byte_arr, format='JPEG')
  File "/usr/local/lib/python3.12/site-packages/PIL/Image.py", line 2568, in save
    save_handler(self, fp, filename)
  File "/usr/local/lib/python3.12/site-packages/PIL/JpegImagePlugin.py", line 642, in _save
    raise OSError(msg) from e
OSError: cannot write mode P as JPEG
2024-08-19 12:51:04.720 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140584923621712] Error handling message: Unknown error (unknown_error) Strato Doumanis from 127.0.0.1 (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.6613.36 Safari/537.36)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/PIL/JpegImagePlugin.py", line 639, in _save
    rawmode = RAWMODE[im.mode]
              ~~~~~~~^^^^^^^^^
KeyError: 'P'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 28, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 793, in handle_execute_script
    script_result = await script_obj.async_run(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1799, in async_run
    return await asyncio.shield(create_eager_task(run.async_run()))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 463, in async_run
    await self._async_step(log_exceptions=False)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 527, in _async_step
    self._handle_exception(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 557, in _handle_exception
    raise exception
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 525, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 764, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 727, in _async_run_long_action
    return await long_task
           ^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2763, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2806, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/__init__.py", line 125, in image_analyzer
    client = await processor.add_images(call.image_entities, call.image_paths, call.target_width, call.include_filename)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 106, in add_images
    base64_image=await self.resize_image(target_width=target_width, image_data=image_data),
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 69, in resize_image
    base64_image = await self._encode_image(img)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 24, in _encode_image
    img.save(img_byte_arr, format='JPEG')
  File "/usr/local/lib/python3.12/site-packages/PIL/Image.py", line 2568, in save
    save_handler(self, fp, filename)
  File "/usr/local/lib/python3.12/site-packages/PIL/JpegImagePlugin.py", line 642, in _save
    raise OSError(msg) from e
OSError: cannot write mode P as JPEG

Additional context

image image image

valentinfrlch commented 2 months ago

I forgot to change the version number in the manifest, which is why it still shows 1.0.3. I have pushed a new release (v1.0.4a, it will show as 1.0.4) to fix this. Can you try reinstalling the new release?

mediacutlet commented 2 months ago

That corrected the version number showing on the integration, so that reduces my confusion a bit. Unfortunately I am still not able to process gifs.

image

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 525, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 764, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 727, in _async_run_long_action
    return await long_task
           ^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2763, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2806, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/__init__.py", line 125, in image_analyzer
    client = await processor.add_images(call.image_entities, call.image_paths, call.target_width, call.include_filename)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 106, in add_images
    base64_image=await self.resize_image(target_width=target_width, image_data=image_data),
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 69, in resize_image
    base64_image = await self._encode_image(img)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 24, in _encode_image
    img.save(img_byte_arr, format='JPEG')
  File "/usr/local/lib/python3.12/site-packages/PIL/Image.py", line 2568, in save
    save_handler(self, fp, filename)
  File "/usr/local/lib/python3.12/site-packages/PIL/JpegImagePlugin.py", line 642, in _save
    raise OSError(msg) from e
OSError: cannot write mode P as JPEG
2024-08-19 15:58:58.832 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139787304414960] Error handling message: Unknown error (unknown_error) Strato Doumanis from 127.0.0.1 (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.6613.36 Safari/537.36)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/PIL/JpegImagePlugin.py", line 639, in _save
    rawmode = RAWMODE[im.mode]
              ~~~~~~~^^^^^^^^^
KeyError: 'P'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 28, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 793, in handle_execute_script
    script_result = await script_obj.async_run(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1799, in async_run
    return await asyncio.shield(create_eager_task(run.async_run()))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 463, in async_run
    await self._async_step(log_exceptions=False)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 527, in _async_step
    self._handle_exception(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 557, in _handle_exception
    raise exception
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 525, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 764, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 727, in _async_run_long_action
    return await long_task
           ^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2763, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2806, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/__init__.py", line 125, in image_analyzer
    client = await processor.add_images(call.image_entities, call.image_paths, call.target_width, call.include_filename)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 106, in add_images
    base64_image=await self.resize_image(target_width=target_width, image_data=image_data),
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 69, in resize_image
    base64_image = await self._encode_image(img)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 24, in _encode_image
    img.save(img_byte_arr, format='JPEG')
  File "/usr/local/lib/python3.12/site-packages/PIL/Image.py", line 2568, in save
    save_handler(self, fp, filename)
  File "/usr/local/lib/python3.12/site-packages/PIL/JpegImagePlugin.py", line 642, in _save
    raise OSError(msg) from e
OSError: cannot write mode P as JPEG
2024-08-19 15:59:32.277 INFO (MainThread) [custom_components.llmvision.request_handlers] Fetching http://192.168.0.229:8123/api/image_proxy/image.weather_radar_new_jersey?token=da4f2ba02e4e1c2331bcb97a5d8a1a95a155fbba122d871737c62fa9028a768b
2024-08-19 15:59:32.289 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Unexpected error for call_service at pos 1: cannot write mode P as JPEG
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/PIL/JpegImagePlugin.py", line 639, in _save
    rawmode = RAWMODE[im.mode]
              ~~~~~~~^^^^^^^^^
KeyError: 'P'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 525, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 764, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 727, in _async_run_long_action
    return await long_task
           ^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2763, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2806, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/__init__.py", line 125, in image_analyzer
    client = await processor.add_images(call.image_entities, call.image_paths, call.target_width, call.include_filename)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 106, in add_images
    base64_image=await self.resize_image(target_width=target_width, image_data=image_data),
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 69, in resize_image
    base64_image = await self._encode_image(img)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 24, in _encode_image
    img.save(img_byte_arr, format='JPEG')
  File "/usr/local/lib/python3.12/site-packages/PIL/Image.py", line 2568, in save
    save_handler(self, fp, filename)
  File "/usr/local/lib/python3.12/site-packages/PIL/JpegImagePlugin.py", line 642, in _save
    raise OSError(msg) from e
OSError: cannot write mode P as JPEG
2024-08-19 15:59:32.294 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [139787304414960] Error handling message: Unknown error (unknown_error) Strato Doumanis from 127.0.0.1 (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.6613.36 Safari/537.36)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/PIL/JpegImagePlugin.py", line 639, in _save
    rawmode = RAWMODE[im.mode]
              ~~~~~~~^^^^^^^^^
KeyError: 'P'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 28, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 793, in handle_execute_script
    script_result = await script_obj.async_run(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1799, in async_run
    return await asyncio.shield(create_eager_task(run.async_run()))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 463, in async_run
    await self._async_step(log_exceptions=False)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 527, in _async_step
    self._handle_exception(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 557, in _handle_exception
    raise exception
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 525, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 764, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 727, in _async_run_long_action
    return await long_task
           ^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2763, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2806, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/__init__.py", line 125, in image_analyzer
    client = await processor.add_images(call.image_entities, call.image_paths, call.target_width, call.include_filename)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 106, in add_images
    base64_image=await self.resize_image(target_width=target_width, image_data=image_data),
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 69, in resize_image
    base64_image = await self._encode_image(img)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 24, in _encode_image
    img.save(img_byte_arr, format='JPEG')
  File "/usr/local/lib/python3.12/site-packages/PIL/Image.py", line 2568, in save
    save_handler(self, fp, filename)
  File "/usr/local/lib/python3.12/site-packages/PIL/JpegImagePlugin.py", line 642, in _save
    raise OSError(msg) from e
OSError: cannot write mode P as JPEG
valentinfrlch commented 2 months ago

Can you try downloading the image to your Home Assistant (use the /www/tmp/ folder) and try with the image path as input? This is the image I tested with: https://radar.weather.gov/ridge/standard/KDIX_0.gif And this would be the path: /config/www/tmp/KDIX_0.gif

mediacutlet commented 2 months ago

Confirmed, locally hosting the gif works. So it seems the issue is isolated to using gifs with "Image Entity"

image

valentinfrlch commented 2 months ago

Thanks for testing! I'll take a look. How did you set up the helper image entity?

mediacutlet commented 2 months ago

Sure thing! Check out the video below -- simple 'image template' helper.

https://github.com/user-attachments/assets/61c4602f-f059-4856-a442-43a9221780e9

mediacutlet commented 2 months ago

For now I'm using the Downloader integration to scrape the latest radar snapshot and drop into a local folder for this integration to pickup. Working splendidly so far @valentinfrlch.

valentinfrlch commented 2 months ago

Thank you very much for the detailed explanation! I'll take a look this.

valentinfrlch commented 2 months ago

Fixed it! The issue was that I only converted to jpg when using local paths and forgot to change the method that is called when using image entities. Either way, gifs should now finally be fully supported in the next release 🚀.

mediacutlet commented 2 months ago

Huzzah! Thanks mate. 👏🏼