[Bug]: Giving Gemini an image whose URL does not end with an extension causes a 500 error

wwwcojp commented 4 days ago

What happened?

Hi there!

I sent a request to Gemini on Litellm proxy with the image in URL format instead of BASE64 format.

In that case, if the image URL ended with an extension, it succeeded without any problem. However, when the image URL did not end with an extension, a 500 error occurred.

Request Format

{
    "model": "gemini-1.5-flash",
    "messages": [
        {
        "role": "user",
        "content": [{"type": "text", "text": "describe this image"},
                    {"type": "image_url", "image_url": {"url": "https://images.pexels.com/photos/1319515/pexels-photo-1319515.jpeg?auto=compress&cs=tinysrgb&w=1260&h=750&dpr=1" }}
                ]
            }
    ],
}

Error Message

{"error":{"message":"litellm.InternalServerError: VertexAIException InternalServerError - Image url not in expected format. Example Expected input - \"image_url\": \"data:image/jpeg;base64,{base64_image}\". Supported formats - ['image/jpeg', 'image/png', 'image/gif', 'image/webp'] ","type":null,"param":null,"code":500}

From the error message, I think the cause is that the following process determines the file type at the end of the URL.

https://github.com/BerriAI/litellm/blob/main/litellm/llms/prompt_templates/factory.py#L666-L678

There are many cases where the URL of an image does not end with an extension, such as a signed URL in S3 or a SAS URL in Azure BLOB Storage. Therefore, we would appreciate it if you could add a process to properly determine the file type in order to combine this with a mechanism to securely publish sensitive data.