Are there some models in ollama can support function calling or object return?

jackmpcollins commented 6 months ago

@chaos369 Looks like ollama does support tool calls, but litellm currently has some bugs with how it is parsing these. I've opened an issue there https://github.com/BerriAI/litellm/issues/3333 When that is fixed you should be able to use ollama models via the LitellmChatModel in magentic by using the "ollama_chat/" prefix (instead of "ollama/") in the model name.

Something like this (based on example from https://github.com/jackmpcollins/magentic/issues/50#issuecomment-1794099320)

from magentic import prompt
from magentic.chat_model.litellm_chat_model import LitellmChatModel

@prompt(
    "Count to 5",
    model=LitellmChatModel("ollama_chat/llama2", api_base="http://localhost:11434")
)
def test() -> list[int]: ...

test()

@knoopx you might be interested in following this issue

chaos369 commented 5 months ago

Thank you.

jackmpcollins commented 5 months ago

I've opened a PR on litellm that should fix this https://github.com/BerriAI/litellm/pull/3469 Just need to bump the litellm dependency version in magentic once that's merged and released.

jackmpcollins commented 5 months ago

@chaos369 @DevAseel I have just published https://github.com/jackmpcollins/magentic/releases/tag/v0.23.0 which enables structured outputs and function calling with ollama 🦙 Depending on what model you use you might need to prompt it to "use the tool" or add more details of the response format in the prompt.

Example

from magentic import prompt
from magentic.chat_model.litellm_chat_model import LitellmChatModel

@prompt(
    "Count to {n}. Use the tool to return in the format [1, 2, 3, ...]",
    model=LitellmChatModel("ollama_chat/llama2", api_base="http://localhost:11434")
)
def count_to(n: int) -> list[int]: ...

count_to(5)
# > [1, 2, 3, 4, 5]

Please let me know if you hit any issues with this. Thanks

chaos369 commented 5 months ago

It seems hardly to return an object. @jackmpcollins

code for testing

wizardlm2=LitellmChatModel("ollama/wizardlm2:7b") llama3 = LitellmChatModel("ollama/llama3:instruct") gpt35=LitellmChatModel(model="openai/gpt-3.5-turbo")

llm_model = wizardlm2

class Superhero(BaseModel): name: str age: int power: str enemies: list[str]

@prompt("Create a Superhero named {name}, use the tool to return a Superhero struct.", model=llm_model) def create_superhero(name: str) -> Superhero: ...

hero = create_superhero("Garden Man")

the result when "llm_model = wizardlm2"

Traceback (most recent call last): File "/home/mac/workspace/projects/mag_hero.py", line 23, in hero = create_superhero("Garden Man") File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/magentic/prompt_function.py", line 81, in call message = self.model.complete( File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/magentic/chat_model/litellm_chat_model.py", line 176, in complete content = next(parse_streamed_tool_calls(response, tool_schemas)) File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/magentic/chat_model/openai_chat_model.py", line 266, in parse_streamed_tool_calls tool_schema = select_tool_schema(first_chunk, tool_schemas) File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/magentic/chat_model/openai_chat_model.py", line 215, in select_tool_schema raise ValueError(msg) ValueError: Unknown tool call: {"id":"call_6618f929-fcd0-4a54-9cfe-48c3dcedf42f","function":{"arguments":"{\"argument_name\": \"Garden Man\", \"power\": \"Ability to control and manipulate plants, flora, and fauna; can also create powerful plant-based weapons.\", \"enemies\": [\"Pollutants\", \"Urban Developers\", \"Invasive Species Villains\"], \"age\": 35}","name":"createGardenMan"},"type":"function","index":0}

the result when "llm_model = llama3"

Traceback (most recent call last): File "/home/mac/workspace/projects/mag_hero.py", line 23, in hero = create_superhero("Garden Man") File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/magentic/prompt_function.py", line 81, in call message = self.model.complete( File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/magentic/chat_model/litellm_chat_model.py", line 154, in complete first_chunk = next(response) File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/litellm/llms/ollama.py", line 297, in ollama_completion_stream raise e File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/litellm/llms/ollama.py", line 256, in ollama_completion_stream status_code=response.status_code, message=response.text File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/httpx/_models.py", line 574, in text content = self.content File "/home/mac/miniconda3/envs/llama/lib/python3.10/site-packages/httpx/_models.py", line 568, in content raise ResponseNotRead() httpx.ResponseNotRead: Attempted to access streaming response content, without having called read().

jackmpcollins commented 5 months ago

@chaos369 I think the ResponseNotRead issue here could be caused by Ollama. Could you make sure it is fully up-to-date and that you have the llama3 model pulled/downloaded.

The first issue is caused by the model returning an invalid function name, createGardenMan instead of return_superhero which is expected by magentic. I've tried several prompts with llama3 to fix this but it is very unreliable. I think the solution will required updates to ollama and litellm. I've opened a new magentic issue to track this https://github.com/jackmpcollins/magentic/issues/207 so please follow that.

jackmpcollins / magentic