[Bug]: Vision + `ollama_chat` not work

[Clad3815]

What happened?

We can't use Vision with ollama_chat, but it's working with ollama.

config.yaml

  - model_name: 'llava:7b'
    litellm_params:
      model: 'ollama_chat/llava:7b'
      api_base: 'http://host.docker.internal:11434'

I use the OpenAI sdk with proxy set to litellm

Relevant log output

16:29:08 - LiteLLM Proxy:ERROR: proxy_server.py:3154 - litellm.proxy.proxy_server.chat_completion(): Exception occured - litellm.APIConnectionError: {"error":"json: cannot unmarshal array into Go struct field Message.messages.content of type string"}
litellm-1  | Traceback (most recent call last):
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 380, in acompletion
litellm-1  |     response = await init_response
litellm-1  |                ^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/llms/ollama_chat.py", line 539, in ollama_acompletion
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/llms/ollama_chat.py", line 479, in ollama_acompletion
litellm-1  |     raise OllamaError(status_code=resp.status, message=text)
litellm-1  | litellm.llms.ollama_chat.OllamaError: {"error":"json: cannot unmarshal array into Go struct field Message.messages.content of type string"}
litellm-1  |  LiteLLM Retried: 2 times, LiteLLM Max Retries: 3
litellm-1  | Traceback (most recent call last):
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 380, in acompletion
litellm-1  |     response = await init_response
litellm-1  |                ^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/llms/ollama_chat.py", line 539, in ollama_acompletion
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/llms/ollama_chat.py", line 479, in ollama_acompletion
litellm-1  |     raise OllamaError(status_code=resp.status, message=text)
litellm-1  | litellm.llms.ollama_chat.OllamaError: {"error":"json: cannot unmarshal array into Go struct field Message.messages.content of type string"}
litellm-1  | 
litellm-1  | During handling of the above exception, another exception occurred:
litellm-1  | 
litellm-1  | Traceback (most recent call last):
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/proxy/proxy_server.py", line 3053, in chat_completion
litellm-1  |     responses = await llm_responses
litellm-1  |                 ^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 645, in acompletion
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 633, in acompletion
litellm-1  |     response = await self.async_function_with_fallbacks(**kwargs)
litellm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2317, in async_function_with_fallbacks
litellm-1  |     raise original_exception
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2162, in async_function_with_fallbacks
litellm-1  |     response = await self.async_function_with_retries(*args, **kwargs)
litellm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2417, in async_function_with_retries
litellm-1  |     raise original_exception
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 2339, in async_function_with_retries
litellm-1  |     response = await original_function(*args, **kwargs)
litellm-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 775, in _acompletion
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/router.py", line 747, in _acompletion
litellm-1  |     response = await _response
litellm-1  |                ^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 1500, in wrapper_async
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 1315, in wrapper_async
litellm-1  |     result = await original_function(*args, **kwargs)
litellm-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 408, in acompletion
litellm-1  |     raise exception_type(
litellm-1  |           ^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 7631, in exception_type
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/utils.py", line 7595, in exception_type
litellm-1  |     raise APIConnectionError(
litellm-1  | litellm.exceptions.APIConnectionError: litellm.APIConnectionError: {"error":"json: cannot unmarshal array into Go struct field Message.messages.content of type string"}
litellm-1  | Traceback (most recent call last):
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/main.py", line 380, in acompletion
litellm-1  |     response = await init_response
litellm-1  |                ^^^^^^^^^^^^^^^^^^^
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/llms/ollama_chat.py", line 539, in ollama_acompletion
litellm-1  |     raise e
litellm-1  |   File "/usr/local/lib/python3.11/site-packages/litellm/llms/ollama_chat.py", line 479, in ollama_acompletion
litellm-1  |     raise OllamaError(status_code=resp.status, message=text)
litellm-1  | litellm.llms.ollama_chat.OllamaError: {"error":"json: cannot unmarshal array into Go struct field Message.messages.content of type string"}
litellm-1  |  LiteLLM Retried: 2 times, LiteLLM Max Retries: 3

Twitter / LinkedIn details

No response

[krrishdholakia]

can you share a sample curl for repro? @Clad3815

[Clad3815]

I'm not using curl, just OpenAI sdk with litellm setup as proxy (On docker).

This config work:

  - model_name: 'llava:7b'
    litellm_params:
      model: 'ollama/llava:7b'
      api_base: 'http://host.docker.internal:11434'

But not this one:

  - model_name: 'llava:7b'
    litellm_params:
      model: 'ollama_chat/llava:7b'
      api_base: 'http://host.docker.internal:11434'

I don't know what I can provide more, any request will result in that

[manojmanivannan]

facing the same issue.

[KuriaMaingi]

My fix was to instead regular ollama not ollama_chat:

litellm --model ollama/llama3.1:latest --drop_params

[wu0up]

I am facing the same issue with the Ollama function call using the Llava model. When I use ollama/llava, I get the error: KeyError: 'name': function_call["name"]. However, when I switch to ollama_chat/llava, I encounter a different error: json: cannot unmarshal array into Go struct field ChatRequest.messages of type string."