Automatic image blob creation doesn't handle RGBA images with JPEG.

[FrostyTheSouthernSnowman]

Description of the bug:

Calling generate_content on a Gemini Pro Vision model returns an error when it receives a PNG image saying KeyError: 'RGBA' which causes another execption saying OSError: cannot write mode RGBA as JPEG. This seems to indicate that PNG is not supported, but according to the Gemini API docs, PNG is a supported MIME type. Note that the png example from that docs page doesn't seem to work. It uses a contents kwarg to generate_content, but that argument doesn't exist. Modifying the code to use the right arguments gives the error google.api_core.exceptions.InvalidArgument: 400 Request contains an invalid argument.

Actual vs expected behavior:

The expected behavior is for this code:

screenshot = get_screen_data()

prompt = "What are your thoughts on this screenshot? I think"

response = model.generate_content(
    [prompt, screenshot], stream=True
)

response.resolve()

print(response.text)

to work successfully. This code was modified from the text from image and text example in the quickstart. Instead, it outputs the KeyError and OSError above. Changing the code to:

screenshot = get_screen_data()

screenshot_data = {
    'mime_type': 'image/png',
    'data': screenshot.tobytes()
}

prompt = "What are your thoughts on this screenshot? I think"

response = model.generate_content(
    [prompt, screenshot_data], stream=True
)

response.resolve()
print(response.text)

Raises a 400 error as described above. This code is modified from that Gemini API Overview

Any other information you'd like to share?

112 is related to this. Specifically, it deals with my second attempt at solving this problem. This issue is about the fact that generate_content doesn't handle PNG by default even though it is supposedly supported.

[Andy963]

it seems that the code in Gemini API Overview is not correct,

model = genai.GenerativeModel('gemini-pro-vision')

cookie_picture = [{
    'mime_type': 'image/png',
    'data': Path('cookie.png').read_bytes()
}]
prompt = "Do these look store-bought or homemade?"

response = model.generate_content(
    model="gemini-pro-vision", # parameter model is no need here
    content=[prompt, cookie_picture]
)
print(response.text)

[FrostyTheSouthernSnowman]

Definitely seems to be the case

[MarkDaoust]

In my tests PNG is working fine.

IDK what your screenshot = get_screen_data() function is.

Can you share a colab that reproducs es the problem?

it seems that the code in Gemini API Overview is not correct,

Thanks, I'm sending a fix for this.

[ya-stack]

Hi, I am trying to read image from https: URL, but it seems to be not working, it's showing below error: ChatGoogleGenerativeAIError: Invalid argument provided to Gemini: 400 Add an image to use models/gemini-pro-vision, or switch your model to a text model.

[FrostyTheSouthernSnowman]

In my tests PNG is working fine.

IDK what your screenshot = get_screen_data() function is.

Can you share a colab that reproducs es the problem?

it seems that the code in Gemini API Overview is not correct,

Thanks, I'm sending a fix for this.

Here's the get_screen_data():

    screen = ImageGrab.grab(bbox=(0, 0, *primary_monitor_dimensions))

    screen = draw_mouse(screen)

    screen = screen.resize((int(screen.size[0] / 2), int(screen.size[1] / 2)))

    if save_screenshot:
        screen.save('screen.png')

    return screen```

ImageGrab comes from PIL.

[github-actions[bot]]

Marking this issue as stale since it has been open for 14 days with no activity. This issue will be closed if no further activity occurs.

[MarkDaoust]

This is caused because the code generates the bytes to send tries to create a JPEG file, but the image is RGBA. Adding a .convert('RGB') before saving it fixes this.

In [13]: model = genai.GenerativeModel(model_name='gemini-pro-vision')

In [14]: model.generate_content([img2, "what's this"])
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File ~/Projects/venv3/lib/python3.11/site-packages/PIL/JpegImagePlugin.py:650, in _save(im, fp, filename)
    649 try:
--> 650     rawmode = RAWMODE[im.mode]
    651 except KeyError as e:

KeyError: 'RGBA'

The above exception was the direct cause of the following exception:

OSError                                   Traceback (most recent call last)
Cell In[14], line 1
----> 1 model.generate_content([img2, "what's this"])

File ~/Projects/generative-ai-python/google/generativeai/generative_models.py:236, in GenerativeModel.generate_content(self, contents, generation_config, safety_settings, stream, tools, tool_config, request_options)
    233 if not contents:
    234     raise TypeError("contents must not be empty")
--> 236 request = self._prepare_request(
    237     contents=contents,
    238     generation_config=generation_config,
    239     safety_settings=safety_settings,
    240     tools=tools,
    241     tool_config=tool_config,
    242 )
    243 if self._client is None:
    244     self._client = client.get_default_generative_client()

File ~/Projects/generative-ai-python/google/generativeai/generative_models.py:139, in GenerativeModel._prepare_request(self, contents, generation_config, safety_settings, tools, tool_config)
    136 else:
    137     tool_config = content_types.to_tool_config(tool_config)
--> 139 contents = content_types.to_contents(contents)
    141 generation_config = generation_types.to_generation_config_dict(generation_config)
    142 merged_gc = self._generation_config.copy()

File ~/Projects/generative-ai-python/google/generativeai/types/content_types.py:293, in to_contents(contents)
    288     except TypeError:
    289         # If you get a TypeError here it's probably because that was a list
    290         # of parts, not a list of contents, so fall back to `to_content`.
    291         pass
--> 293 contents = [to_content(contents)]
    294 return contents

File ~/Projects/generative-ai-python/google/generativeai/types/content_types.py:256, in to_content(content)
    254     return content
    255 elif isinstance(content, Iterable) and not isinstance(content, str):
--> 256     return protos.Content(parts=[to_part(part) for part in content])
    257 else:
    258     # Maybe this is a Part?
    259     return protos.Content(parts=[to_part(content)])

File ~/Projects/generative-ai-python/google/generativeai/types/content_types.py:256, in <listcomp>(.0)
    254     return content
    255 elif isinstance(content, Iterable) and not isinstance(content, str):
--> 256     return protos.Content(parts=[to_part(part) for part in content])
    257 else:
    258     # Maybe this is a Part?
    259     return protos.Content(parts=[to_part(content)])

File ~/Projects/generative-ai-python/google/generativeai/types/content_types.py:224, in to_part(part)
    220     return protos.Part(function_response=part)
    222 else:
    223     # Maybe it can be turned into a blob?
--> 224     return protos.Part(inline_data=to_blob(part))

File ~/Projects/generative-ai-python/google/generativeai/types/content_types.py:164, in to_blob(blob)
    162     return blob
    163 elif isinstance(blob, IMAGE_TYPES):
--> 164     return image_to_blob(blob)
    165 else:
    166     if isinstance(blob, Mapping):

File ~/Projects/generative-ai-python/google/generativeai/types/content_types.py:89, in image_to_blob(image)
     87 if PIL is not None:
     88     if isinstance(image, PIL.Image.Image):
---> 89         return pil_to_blob(image)
     91 if IPython is not None:
     92     if isinstance(image, IPython.display.Image):

File ~/Projects/generative-ai-python/google/generativeai/types/content_types.py:79, in pil_to_blob(img)
     77     mime_type = "image/png"
     78 else:
---> 79     img.save(bytesio, format="JPEG")
     80     mime_type = "image/jpeg"
     81 bytesio.seek(0)

File ~/Projects/venv3/lib/python3.11/site-packages/PIL/Image.py:2439, in Image.save(self, fp, format, **params)
   2436         fp = builtins.open(filename, "w+b")
   2438 try:
-> 2439     save_handler(self, fp, filename)
   2440 except Exception:
   2441     if open_fp:

File ~/Projects/venv3/lib/python3.11/site-packages/PIL/JpegImagePlugin.py:653, in _save(im, fp, filename)
    651 except KeyError as e:
    652     msg = f"cannot write mode {im.mode} as JPEG"
--> 653     raise OSError(msg) from e
    655 info = im.encoderinfo
    657 dpi = [round(x) for x in info.get("dpi", (0, 0))]