crewAIInc / crewAI

Framework for orchestrating role-playing, autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks.
https://crewai.com
MIT License
20.63k stars 2.86k forks source link

[BUG] CrewAi Dalli Tool Issue #1522

Closed yuriiIykyknow closed 14 hours ago

yuriiIykyknow commented 2 days ago

Description

Agent provides wrong input value to Dalli tool. Also, crewAi can't provide right arguments to Dalli tool because args_schema is wrong. Excpect 'image_description' but got 'description' key

Steps to Reproduce

  1. uv run main.py

Expected behavior

Get link to image)

Screenshots/Code snippets

from crewai import Agent, Task, Crew
from crewai_tools import DallETool
from asyncio import run

async def dalle_test():
    dalle_tool = DallETool(
        model="dall-e-3",
        size="1024x1024",
        quality="standard",
        n=1
        )
    agent = Agent(
        role='Artist',
        goal='Draw a cat with wings',
        backstory="""You're a senior artist at a game development company. You've been tasked with creating a concept art for a new game character. The character is a cat with wings, and it should look cute and friendly. The art will be used to pitch the character to the game designers and developers.""",
        tools=[dalle_tool],
    )
    task = Task(
        description='draw a cat with wings',
        agent=agent,
        expected_output='give a link to the image start with http:// ot https://',
        )
    crew = Crew(
        agents=[agent],
        tasks=[task],
        verbose=True,
        )
    crew_output = crew.kickoff()
    return crew_output

async def main():
    flow = await dalle_test()
    result = await flow.kickoff()
    print(f"Generated fun fact: {result}")

if __name__ == "__main__":
    # Run both examples
    run(main())

Operating System

Windows 11

Python Version

3.12

crewAI Version

0.76.2

crewAI Tools Version

0.13.2

Virtual Environment

Venv

Evidence

Agent: Artist Thought: Thought: I need to ensure I am using the right key name in the Action Input for the Dall-E Tool. I should properly encapsulate the attributes in the correct format. Using tool: Dall-E Tool Tool Input: "{\"name\": \"Cat with Wings\", \"description\": \"A cute and friendly cat with colorful butterfly-like wings, big sparkling eyes, and a fluffy tail. The cat is posed playfully, and the background is whimsical and colorful, adding to its charm.\", \"args_schema\": {}, \"return_direct\": true}" Tool Output: Error: the Action Input is not a valid key, value dictionary.

Agent: Artist Thought: Thought: I have been using repeated descriptions without success. I'll simplify the action to just focus on the essential details for creating the cat with wings. Using tool: Dall-E Tool Tool Input: "{\"name\": \"Cat with wings\", \"description\": \"A cute cat with vibrant, fluffy wings, looking friendly and playful. The cat is sitting with a joyful expression, set against a cheerful, whimsical background.\", \"args_schema\": \"BaseModel\"}" Tool Output: Image description is required.

Possible Solution

  1. Agent sends wrong format to tools. (assume issue in tool_usage.py in crewai source)
  2. Image description is required." (assume issue in dalle_tool.py how we grab args from args_scheme)

Additional context

None

sorin-costea commented 2 days ago

I get this in different configurations with the tools I use: PDFSearchTool always works, DOCXSearchTool always fails, and SerperSearchTool sometimes works sometimes not. When it works, the agents send the correct inputs:

## Tool Input:
"{\"search_query\": \"Whatever terms\"}"

And always when failing, the wrong tool input looks the same as yours:

## Tool Input:
"{\"name\": \"Something relevant\", \"description\": \"Some text to be analyzed.\", \"args_schema\": \"None\"}"

No idea where this is decided, when to use the good search_query and when the wrong name/description/args_schema, but tools definitely don't like the latter.

PS: maybe related to https://github.com/crewAIInc/crewAI/issues/1344 ?

yuriiIykyknow commented 2 days ago

When I set up CrewAI locally from the GitHub repository, the Dalli tool worked as expected. So, I assume the work on that is done but has not been pushed to PyPI. i see changes in agent.py image

bhancockio commented 2 days ago

@yuriiIykyknow Thank you so much for reporting this!

Looking into this right now. I think we'll be able to push the new version that will address this issue later tonight or first thing tomorrow.

I'll update you and close this as soon as the new version get's cut!

c0dezli commented 2 days ago

@yuriiIykyknow I think this is related to what i found in #1511, which should be addressed in new releases

nate-walter commented 1 day ago

Yeah I'm seeing this bug in my Crew

bhancockio commented 14 hours ago

Hey @yuriiIykyknow! The new version just dropped. Please update to the latest version of crewAI (0.76.9) and this issue should go away.

If anything else comes up, please let me know!