Unknown Error when trying to use GIF (Still frame) image

[mediacutlet]

Bug Description

I am able to get responses back when using jpg or png images but gifs (non-animated) throw an unknown error. I am attempting to include the image via an image helper entity. I can use this same methodology with jpg and png but not gif.

Public URL of the gif which the sensor updates from: https://radar.weather.gov/ridge/standard/KDIX_0.gif

HA Version: 2024.8.2 Integration Version: 1.0.3

Service Call

action: llmvision.image_analyzer
data:
  provider: OpenAI
  model: gpt-4o-mini
  include_filename: false
  target_width: 1280
  detail: high
  max_tokens: 100
  temperature: 0.5
  image_entity:
    - image.weather_radar_new_jersey
  message: Describe this image

Logs

  File "/config/custom_components/llmvision/__init__.py", line 125, in image_analyzer
    client = await processor.add_images(call.image_entities, call.image_paths, call.target_width, call.include_filename)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 102, in add_images
    base64_image=await self.resize_image(target_width=target_width, image_data=image_data),
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 65, in resize_image
    base64_image = await self._encode_image(img)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 24, in _encode_image
    img.save(img_byte_arr, format='JPEG')
  File "/usr/local/lib/python3.12/site-packages/PIL/Image.py", line 2568, in save
    save_handler(self, fp, filename)
  File "/usr/local/lib/python3.12/site-packages/PIL/JpegImagePlugin.py", line 642, in _save
    raise OSError(msg) from e
OSError: cannot write mode P as JPEG
2024-08-18 18:29:42.632 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140187505528832] Error handling message: Unknown error (unknown_error) Strato Doumanis from 127.0.0.1 (Mozilla/5.0 (iPhone; CPU iPhone OS 17_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2024.7 (io.robbie.HomeAssistant; build:2024.730; iOS 17.5.1) Mobile/HomeAssistant, like Safari)
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/PIL/JpegImagePlugin.py", line 639, in _save
    rawmode = RAWMODE[im.mode]
              ~~~~~~~^^^^^^^^^
KeyError: 'P'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 28, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 793, in handle_execute_script
    script_result = await script_obj.async_run(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1799, in async_run
    return await asyncio.shield(create_eager_task(run.async_run()))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 463, in async_run
    await self._async_step(log_exceptions=False)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 527, in _async_step
    self._handle_exception(
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 557, in _handle_exception
    raise exception
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 525, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 764, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 727, in _async_run_long_action
    return await long_task
           ^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2763, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2806, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/__init__.py", line 125, in image_analyzer
    client = await processor.add_images(call.image_entities, call.image_paths, call.target_width, call.include_filename)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 102, in add_images
    base64_image=await self.resize_image(target_width=target_width, image_data=image_data),
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 65, in resize_image
    base64_image = await self._encode_image(img)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/llmvision/media_handlers.py", line 24, in _encode_image
    img.save(img_byte_arr, format='JPEG')

Additional context

Error: Failed to perform the action llmvision.image_analyzer. Unknown error

[valentinfrlch]

You are right, gifs are not supported right now but I'm going to add that in the next update. Looks like you'd have a great use case for it!

[mediacutlet]

Awesome! Thanks for your work on this! 😄 🌩️

[valentinfrlch]

The fix was really easy, gifs are now converted to jpg. Should be live in a couple hours. Also, I really like the idea with the weather map. Would it be ok if I add this as an example to the wiki?

[mediacutlet]

Absolutely! Happy to contribute @valentinfrlch. Thank you for implementing gif support so quickly.

One thing that might be interesting, expanding upon the weather radar use case, would be animated gif support. I'm sure that will be more involved since converting to jpg won't work with animation (unless maybe it converted each frame of the animation to jpg?). But animation would give the LLM quite a bite more information -- directionality of the radar features, speed, etc.

[valentinfrlch]

Thanks! Yes animated GIFs would be a little more involved but still very possible. The integration already supports mp4 (via the video_analyzer) so support for animated GIFs shouldn't be too difficult either. I'll add that to the roadmap!

[mediacutlet]

Thanks for adding animated gif support @valentinfrlch I'll be experimenting with this later today and will report back any issues!