search

BerriAI / litellm

Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
https://docs.litellm.ai/docs/
Other
12.72k stars 1.48k forks source link

[Bug]: Image URL disappeared from litellm to localai #2983

Closed thiner closed 5 months ago

thiner commented 5 months ago

What happened?

I have LocalAI as the backend which serves a VL model. LocalAI supports OpenAI gpt-4-vision compatible completions API. I tested the API locally with below request:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "gpt-4-vision-preview", "messages": [{"role": "user", "content": [{"type":"text", "text": "Describe the image?"}, {"type": "image_url", "image_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg" }}], "temperature": 0.2}]}'

The backend log shows that it compiles the image URL into <img>[local file path]</image> based on chat template setting. This is the normal behavior, and the API returns correct answer.

8:44AM DBG GRPC(qwen-vl-chat-int4-127.0.0.1:34155): stderr Prompt: user:<img>/tmp/vl-1712911492417.jpg</img>,Describe the image?
8:44AM DBG GRPC(qwen-vl-chat-int4-127.0.0.1:34155): stderr

But when I setup LiteLLM to proxy the vision request to this LocalAI API, the image url disappeared, LocalAI log as below, the <image> tag is not there.

 9:16AM DBG Prompt (before templating): user:describe the image

 9:16AM DBG Template found, input modified to: Instruct: user:describe the image

Relevant log output

No response

Twitter / LinkedIn details

No response

thiner commented 5 months ago

Sorry, it's not a bug of LiteLLM.