giandre
commented
1 month ago I have similar issues with all my previous code with crewai. I ended up sticking with 0.41.0 version so things will continue to work. I have no idea how to fix it.
Closed manili closed 1 month ago
giandre
commented
1 month ago I have similar issues with all my previous code with crewai. I ended up sticking with 0.41.0 version so things will continue to work. I have no idea how to fix it.
strikeoncmputrz
commented
1 month ago I am encountering the same issue with gpt-4o being called despite defining an LLM in my crew method. The only working Agent is my manager_llm. I assume that it is now necessary to define the LLM on a per agent basis. Is that correct?
@crew
def crew(self) -> Crew:
"""Creates the WriteEssay crew"""
return Crew(
agents=self.agents, # Automatically created by the @agent decorator
tasks=self.tasks, # Automatically created by the @task decorator
llm="openai/LoneStriker_Hermes-3-Llama-3.1-70B-6.0bpw-h6-exl2_main",
function_calling_llm="openai/LoneStriker_Hermes-3-Llama-3.1-70B-6.0bpw-h6-exl2_main",
manager_llm="openai/LoneStriker_Hermes-3-Llama-3.1-70B-6.0bpw-h6-exl2_main",
process=Process.hierarchical,
#process=Process.sequential,
#planning=True,
planning_llm="openai/LoneStriker_Hermes-3-Llama-3.1-70B-6.0bpw-h6-exl2_main",
memory=True,
verbose=True,
share_crew=False,
output_log_file=get_log_file_name(),
# Need to set embedder based on what is being hosted by Infinity-Embeddings
embedder={
"provider": "openai",
"config":{
"model": 'BAAI/bge-small-en-v1.5'
}
}
)
theCyberTech
commented
1 month ago 
olafgeibig
commented
1 month ago I must also chime in here. Making LiteLLM the only way to provide an LLM is a very poor decision for multiple reasons.
First of all its a huge breaking change. All my crews don't work anymore. Okay this can be solved with some editing monkey job, but the worst is that it seems I can't use JSON output with models other than OpenAI via their OpenAI compatible API.
With the previous version I could use output_json with deepseek-chat on tasks. I just need to used an OpenAI chat client configured with deepseek like that:
llm = ChatOpenAI(
model="deepseek-chat",
api_key=DEEPSEEK_API_KEY,
base_url="https://api.deepseek.com/v1",
temperature=0.0
)
note_task = Task(
description="shortened for brevity",
agent=writer_agent,
expected_output="a note that is the filled template",
output_json=Note,
output_file="note.json",
) This code results in valid JSON file. With the new version I managed to use deepseek-chat with the OpenAI API, but the JSON serialisation gives an error now.
load_dotenv()
os.environ["OPENAI_API_KEY"] = os.getenv("DEEPSEEK_API_KEY")
os.environ["OPENAI_API_BASE"] = 'https://api.deepseek.com/v1/'
writer_agent = Agent(
role='Note writer',
goal='Write personal notes using a specified template.',
backstory='I am a note writer with a great skill to write personal notes.',
verbose=True,
llm="openai/deepseek-chat",
allow_delegation=False,
tools=[]
)
note_task = Task(
description="shortened for brevity",
agent=writer_agent,
expected_output="a note that is the filled template",
output_json=Note,
)Error:
pydantic_core._pydantic_core.ValidationError: 1 validation error for TaskOutput
json_dict
Input should be a valid dictionary [type=dict_type, input_value='```json\n{\n "title":...g and support."\n}\n```', input_type=str]
For further information visit https://errors.pydantic.dev/2.8/v/dict_typeUsing llm="deepseek/deepsekk-chat" also does not support JSON output. It seems it doesn't use deepseeks JSON output of their propretary API. Same Pydantic error because the json is surrounded by markdoiwn quotes, means no JSON mode.
Moreover there is now no control over LLM parameters like temperature anymore. Why not leaving the exiting llm parameter and add a new one for LiteLLM like llm_name?
Update: LiteLLM doesn't even support function calling for Anthropic models: https://docs.litellm.ai/docs/completion/input
strikeoncmputrz
commented
1 month ago Thanks for sharing the applicable documentation. Even when assigning all my agents an explicit LLM Crew was still attempting to use GPT-4o. I will also set the environment variable I guess.
joaomdmoura
commented
1 month ago Hey there, catching up to this! This is great feedback!
Good point @olafgeibig on maybe support multiple ways to provide an LLM, and honestly that will be way easier to do now that we removed langchain!
Other than LangChain and LLama-Index, is that other major providers we should consider adding that we might be missing out?
olafgeibig
commented
1 month ago @joaomdmoura good to know that you removed LangChain. Now I can turn off the LangChain instrumentor in my tracing. I must admit that I didn't read the changelog. I see, so you replaced the LangChain LLM support with LiteLLM. Now I understand. I agree that LiteLLM was probably the best option to support many LLMs without building on top of a framework.
Regarding the number of supported LLM APIs there is probably nothing comparable to LangChain and LlamaIndex. I find the new ell framework a very fresh approach to developing with LLMs, but it's very young and only supports OpenAI and Anthropic. https://docs.ell.so/index.html
There are many model providers, routers like openrouter or self hosted LLMs that offer an OpenAI compatible API. For instance LiteLLM is not utliizing the structured output and function calling of DeepSeek's proprietary API, but DeepSeek's OpenAI compatible endpoint is offering that. It would be some relief if crewAI would at least allow using the OpenAI client for crews and agents. That would be probably the easiest way to ease the pain expressed here in this issue.
olafgeibig
commented
1 month ago Update. I managed to make it work with the LiteLLM solution for OpenAI compatible APIs.
Additionally I needed to add the JSON schema of the Pydantic object to the task description. Somehow I though that this happening already under the hood of crewAI. At least deepseek-chat needs that to get it right.
load_dotenv()
os.environ["OPENAI_API_KEY"] = os.getenv("DEEPSEEK_API_KEY")
os.environ["OPENAI_API_BASE"] = 'https://api.deepseek.com/v1'
writer_agent = Agent(
role='Note writer',
goal='Write personal notes using a specified template.',
backstory='I am a note writer with a great skill to write personal notes.',
verbose=True,
llm="openai/deepseek-chat",
allow_delegation=False,
tools=[]
)
note_task = Task(
description=f"""
Create a note about the resource Pydantic using the note template.
The note shall contain a general section explaining the resource and its concepts and a multiple themed section with important links to websites.
Adhere to this schema: {Note.model_json_schema()}
""",
agent=writer_agent,
expected_output="a note that is the filled template",
output_pydantic=Note,
)
crew = Crew(
agents=[writer_agent],
tasks=[note_task],
verbose=True
)
crew_output = crew.kickoff()
print(f"Crew: {crew_output}")
print(f"Pydantic: {note_task.output.pydantic}")Still I don't like about this solution that I need to overwrite the OPENAI_API_KEY in the environment, but at least it works.
BTW, the tasks doc looks a bit weird and contains contradictory information regarding task parameters: https://docs.crewai.com/core-concepts/Tasks/
Kingbadger3d
commented
1 month ago Why, Just Why?. There was no need to make this change to LiteLLM so breaking, yet you did it anyway. Yet Langchain worked as expected and had more features? If changing LLM system why didnt you at least go for something more production oriented like vLLM?. odd decisions, that mean we all have to rewrite huge amounts to just get back to a working condition to have less features?.
theCyberTech
commented
1 month ago Why, Just Why?. There was no need to make this change to LiteLLM so breaking, yet you did it anyway. Yet Langchain worked as expected and had more features? If changing LLM system why didnt you at least go for something more production oriented like vLLM?. odd decisions, that mean we all have to rewrite huge amounts to just get back to a working condition to have less features?.
Curious as to why you think there was no need o make this change?
LiteLLM and vLLM do not have the same main purpose so that so it would not make sense to do that. You can integrate vLLM through LiteLLM already.
Also there is nothing stopping you from using Langchain / llamaindex still.
Let me know if you need help with anything
joaomdmoura
commented
1 month ago @olafgeibig This is a good suggestion, we can still add support to LangChain Model definition without need to bring back their executor now, so this is good, I'm already working on it.
Hey @Kingbadger3d appreciate the messages, I'm sorry you are experiment so much pain from the upgrade. If you want, it might be easier to hold on upgrading while we bring more providers including langchain for model connections. We needed to remove this because langchain was slowing down our ability to add some major new features and would break on some of their updates.
TarcisioAmaral
commented
1 month ago Can someone help me out? man i feel lost. i using CrewAI for build a crew with 'crew.py', 'main.py', 'agents.yaml' and 'tasks.yaml' to configurate and put what do u want, but a get only errors, i read documentation but still i don't get it how realy works. I start with crewai about 2 mounth, i already run some crews but after this version i get lost. there is some model, workflow with this new version and LiteLLM? And some visual feedback, i tink that could be better understanding to the documentarion and also a workflow for the people like me who jump into CrewAI now
joaomdmoura
commented
1 month ago Hey @TarcisioAmaral how much can you share about specific errors you are getting and your specific setup?
This doc page should give you instructions to get a working setup.
The major thing changing in this version was the we removed the LangChain Agent Executor, allowing us to write our own, what gave us more control and the ability to build features we were not able before, so all the remaining features should still work the same, but I would love to dig into specific problems to learn if there are bugs we should fix.
The main change required in your actual code is that now instead of passing the LangChain model definition to your agents you could pass a string like gpt-4o-mini and it will use LiteLLM behind the scenes to do the LLM calls. We will bring back support for the LangChain models without bringing back their executor though to make it easier for people to use the new versions, migrating from older ones in existing projects.
TarcisioAmaral
commented
1 month ago Opa @joaomdmoura. Shure. Is "UnicodeDecodeError", i got but the Output of the terminal here below. Few friends and i pass throught the same problem, we learning crewai with Luan(Brazilian Power hehe). If u can help me gonna be amazing. Gonna read this Doc learn more about.
File "
joaomdmoura
commented
1 month ago Hey folks, spent the day today working on a new LLM class that will be introduced in the next version to address all the concerns in this issue, open for feedback before we cut the verison.
New docs, still not live: (https://github.com/crewAIInc/crewAI/blob/main/docs/core-concepts/LLMs.md) Some high level description bellow:
By default, crewAI uses the gpt-4o-mini model.
It uses environment variables if no LLM is specified:
OPENAI_MODEL_NAME (defaults to "gpt-4o-mini" if not set)OPENAI_API_BASEOPENAI_API_KEYAiming at simplicity and supporting who update to version 0.61.0
agent = Agent(llm="gpt-4o", ...)List of more providers.
from crewai import LLM
llm = LLM(model="gpt-4", temperature=0.7)
agent = Agent(llm=llm, ...)This new LLM class allow you to use an big array of attributes.
| Parameter | Type | Description |
|---|---|---|
model |
str | The name of the model to use (e.g., "gpt-4", "gpt-3.5-turbo", "ollama/llama3.1", more providers) |
timeout |
float, int | Maximum time (in seconds) to wait for a response |
temperature |
float | Controls randomness in output (0.0 to 1.0) |
top_p |
float | Controls diversity of output (0.0 to 1.0) |
n |
int | Number of completions to generate |
stop |
str, List[str] | Sequence(s) to stop generation |
max_tokens |
int | Maximum number of tokens to generate |
presence_penalty |
float | Penalizes new tokens based on their presence in the text so far |
frequency_penalty |
float | Penalizes new tokens based on their frequency in the text so far |
logit_bias |
Dict[int, float] | Modifies likelihood of specified tokens appearing in the completion |
response_format |
Dict[str, Any] | Specifies the format of the response (e.g., {"type": "json_object"}) |
seed |
int | Sets a random seed for deterministic results |
logprobs |
bool | Whether to return log probabilities of the output tokens |
top_logprobs |
int | Number of most likely tokens to return the log probabilities for |
base_url |
str | The base URL for the API endpoint |
api_version |
str | The version of the API to use |
api_key |
str | Your API key for authentication |
Example:
llm = LLM(
model="gpt-4",
temperature=0.8,
max_tokens=150,
top_p=0.9,
frequency_penalty=0.1,
presence_penalty=0.1,
stop=["END"],
seed=42,
base_url="https://api.openai.com/v1",
api_key="your-api-key-here"
)
agent = Agent(llm=llm, ...)crewAI supports using Ollama for running open-source models locally:
ollama run llama2agent = Agent(
llm=LLM(model="ollama/llama3.1", base_url="http://localhost:11434"),
...
)You can change the base API URL for any LLM provider by setting the base_url parameter:
llm = LLM(
model="custom-model-name",
base_url="https://api.your-provider.com/v1",
api_key="your-api-key"
)
agent = Agent(llm=llm, ...)@TarcisioAmaral your problem seems to be unrelated, but looking at the error message, it indicates a problem with character encoding when trying to read a YAML file.
In an attempt to fix it I push a change to master that forcer the utf-8 encoding and it will be live in the next version, it should help with it.
But I also recommend checking the YAML file for any unusual or non-printable characters. You might need to remove or replace problematic characters.
olafgeibig
commented
1 month ago @joaomdmoura I think this is a very good approach. It will return full control over the LLM client parameters to developers. I think this is an easy to apply change that will deliver great benefits.
Just one more question. I did not read all of the related code but I noticed Converter.is_gpt() is used to decide if certain advanced features of OpenAI models are used by crewAI. I think this simple check of the model name does exclude models with an OpenAI compatible API that also offer these advanced features.
A much more fine-grained check would be possible by inspecting LiteLLM's supported model features provided by a LiteLLM function:
litellm.get_supported_openai_params("openrouter/qwen/qwen-2.5-72b-instruct") Example response:
['temperature', 'top_p', 'frequency_penalty', 'presence_penalty', 'repetition_penalty', 'seed', 'max_tokens', 'logit_bias', 'logprobs', 'top_logprobs', 'response_format', 'stop', 'tools', 'tool_choice']Good call, def something we could add, I think the main thing that is used for is to know if it should use actually tool calling or not but will do a pass on it
joaomdmoura
commented
1 month ago new version is out v0.63.1 env vars should work again, also ability to control all different attribute for the LLM using the LLM class, if this is still an issue let us know and we can re-open it :)
Description
I'm using Colab to test crewAI and after upgrading to 0.60.0 using the following code:
The following code does not work anymore:
The code was working perfectly fine before upgrading to this version (tonight).
Steps to Reproduce
N/A
Expected behavior
N/A
Screenshots/Code snippets
Operating System
Other (specify in additional context)
Python Version
3.10
crewAI Version
0.60.0
crewAI Tools Version
0.60.0
Virtual Environment
Venv
Evidence
I'm encountering the following error:
Possible Solution
Downgrade to the previous version.
Additional context
N/A