[Bug]: gpt-4o error when analysis image

[Undertone0809]

What happened?

A bug happened when I use gpt-4o analysis image

Code:

from litellm import completion

messages=[
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {
                "type": "image_url",
                "image_url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg",
            },
        ],
    }
]

response = completion(model="gpt-4o", messages=messages)
print(response)

Relevant log output

D:\Application\Conda\envs\pne\python.exe D:\Projects\CogitAGI\promptulate\private\main.py 

Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Traceback (most recent call last):
  File "D:\Application\Conda\envs\pne\lib\site-packages\litellm\llms\openai.py", line 426, in completion
    raise e
  File "D:\Application\Conda\envs\pne\lib\site-packages\litellm\llms\openai.py", line 385, in completion
    response = openai_client.chat.completions.create(**data, timeout=timeout)  # type: ignore
  File "D:\Application\Conda\envs\pne\lib\site-packages\openai\_utils\_utils.py", line 277, in wrapper
    return func(*args, **kwargs)
  File "D:\Application\Conda\envs\pne\lib\site-packages\openai\resources\chat\completions.py", line 590, in create
    return self._post(
  File "D:\Application\Conda\envs\pne\lib\site-packages\openai\_base_client.py", line 1240, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "D:\Application\Conda\envs\pne\lib\site-packages\openai\_base_client.py", line 921, in request
    return self._request(
  File "D:\Application\Conda\envs\pne\lib\site-packages\openai\_base_client.py", line 1020, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid type for 'messages[0].content[1].image_url': expected an object, but got a string instead.", 'type': 'invalid_request_error', 'param': 'messages[0].content[1].image_url', 'code': 'invalid_type'}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Application\Conda\envs\pne\lib\site-packages\litellm\main.py", line 1065, in completion
    raise e
  File "D:\Application\Conda\envs\pne\lib\site-packages\litellm\main.py", line 1038, in completion
    response = openai_chat_completions.completion(
  File "D:\Application\Conda\envs\pne\lib\site-packages\litellm\llms\openai.py", line 432, in completion
    raise OpenAIError(status_code=e.status_code, message=str(e))
litellm.llms.openai.OpenAIError: Error code: 400 - {'error': {'message': "Invalid type for 'messages[0].content[1].image_url': expected an object, but got a string instead.", 'type': 'invalid_request_error', 'param': 'messages[0].content[1].image_url', 'code': 'invalid_type'}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Projects\CogitAGI\promptulate\private\main.py", line 17, in <module>
    response = completion(model="gpt-4o", messages=messages)
  File "D:\Application\Conda\envs\pne\lib\site-packages\litellm\utils.py", line 3229, in wrapper
    raise e
  File "D:\Application\Conda\envs\pne\lib\site-packages\litellm\utils.py", line 3123, in wrapper
    result = original_function(*args, **kwargs)
  File "D:\Application\Conda\envs\pne\lib\site-packages\litellm\main.py", line 2238, in completion
    raise exception_type(
  File "D:\Application\Conda\envs\pne\lib\site-packages\litellm\utils.py", line 9353, in exception_type
    raise e
  File "D:\Application\Conda\envs\pne\lib\site-packages\litellm\utils.py", line 8120, in exception_type
    raise BadRequestError(
litellm.exceptions.BadRequestError: OpenAIException - Error code: 400 - {'error': {'message': "Invalid type for 'messages[0].content[1].image_url': expected an object, but got a string instead.", 'type': 'invalid_request_error', 'param': 'messages[0].content[1].image_url', 'code': 'invalid_type'}} 
Model: gpt-4o
API Base: https://api.openai.com
Messages: [{'role': 'user', 'content': [{'type': 'text', 'text': "What's in this image?"}, {'type': 'image_url

Process finished with exit code 1

### Twitter / LinkedIn details

_No response_

[krrishdholakia]

@Undertone0809 i think the openai sample code is out of data. Same request fails on openai client as well.

This works for me (on litellm + openai sdk)

from litellm import completion

response = completion(
            model="gpt-4o",
            messages=[
                {
                    "role": "user",
                    "content": [
                        {"type": "text", "text": "Whats in this image?"},
                        {
                            "type": "image_url",
                            "image_url": {
                                "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
                            },
                        },
                    ],
                }
            ],
        )
print(response)

[krrishdholakia]

their api reference also says the content is a list of objects -