Inconsistent output_json from Agents Using Ollama

[andresperezrobinson]

Description: Hello, I'm trying to get a consistent JSON output from agents by setting a pydantic model with fields to fill in. Although the content of my output is correct, the JSON output is not working as expected. I've summarized my code and "unwanted" results below:

import os
from crewai import Agent, Task, Crew
from langchain_community.llms import Ollama
from pydantic import BaseModel

# Environment variables if needed
os.environ["OPENAI_API_BASE"] = "http://localhost:11434"
os.environ["OPENAI_MODEL_NAME"] = "llama3"
os.environ["OPENAI_API_KEY"] = ""

llm = Ollama(
    model="llama3",
    base_url="http://localhost:11434",
    temperature=0
)

assistant = Agent(
    role='Helpful assistant',
    goal='Follow commands',
    backstory='Expert on the NFL',
    llm=llm
)

class EventDetails(BaseModel):
    city: str
    teams_playing: str
    winner: str

superbowl_info = Task(
    description="Give me information about the last Super Bowl",
    expected_output="JSON format with keys: {city, teams_playing, winner}",
    output_json=EventDetails,
    output_file="event_details.json",
    agent=assistant
)

crew = Crew(
    agents=[assistant],
    tasks=[superbowl_info],
    verbose=2
)

result = crew.kickoff()

Unwanted Output:

Failed to convert text into JSON, error: 'name'. Using raw output instead.

[Helpful assistant] Task output: {
"city": "Tampa",
"teams_playing": ["Kansas City Chiefs", "Tampa Bay Buccaneers"],
"winner": "Tampa Bay Buccaneers"
}

Please note that this answer is in the exact JSON format with the required keys: city, teams_playing, and winner. I've double-checked to ensure it's accurate and complete. My job depends on it, indeed!

Steps Taken to Resolve:

• I've tried being overly specific in the prompt to avoid explanations in the output and just get the JSON.
• Extracting the JSON portion from the raw output, but this approach is unreliable when scaling and not optimal.

Issue: Is there any idea why this is happening? Could it be specific to Llama3? What recommendations are there for resolving this?

[gadgethome]

What happens with a different model such as gpt-4 mini? or different Ollama one?

[andresperezrobinson]

What happens with a different model such as gpt-4 mini? or different Ollama one?

I haven't tried other models, but looking online and at previous issues raised, it appears that there is no issue using gpt, and issues come from using other models. I was hoping someone had a different while using something else than gpt.

[zinyando]

What happens with a different model such as gpt-4 mini? or different Ollama one?

I haven't tried other models, but looking online and at previous issues raised, it appears that there is no issue using gpt, and issues come from using other models. I was hoping someone had a different while using something else than gpt.

Try using more powerful models eg Llama 3.1 70B on Groq to see if you get more consistent output. From my experience Ollama model powered agents need a lot of massaging and prompt engineering to be accurate and consistent.

[andresperezrobinson]

What happens with a different model such as gpt-4 mini? or different Ollama one?

I haven't tried other models, but looking online and at previous issues raised, it appears that there is no issue using gpt, and issues come from using other models. I was hoping someone had a different while using something else than gpt.

Try using more powerful models eg Llama 3.1 70B on Groq to see if you get more consistent output. From my experience Ollama model powered agents need a lot of massaging and prompt engineering to be accurate and consistent.

I understand this model might not be as powerful, however, I would expect the JSON output to be tackled by the crewai framework, and not dependent on the model's power.