search

majacinka / crewai-experiments

Experiments with local as well as models available through an api
779 stars 189 forks source link

What is the conclusion then ? #9

Open Docteur-RS opened 1 month ago

Docteur-RS commented 1 month ago

Have you found at least one LLM capable of calling tools ? Because I've tested mistral, dolphin-mixtral and llam3:8B and all of them are lost !

If I read correctly your benchmark none of your LLMs achieved it either.

So now what ? We dump CrewAI ? XD Honnestly I don't see the point of using a framework that can't call tools.

Docteur-RS commented 1 week ago

Found a model that works well with llamaindex : mistral:7b-instruct-v0.3. It has been fine-tunned to make function calling possible.
Haven't tried it with crewai because I think that the framework's internal prompts are not that great and all local models will struggle with it. I think I'll stay with more predictable and lightweight solutions like llamaindex and langgraph.

But if someone does try crewAI with the Mistral V0.3 should post the results here.

Docteur-RS commented 1 week ago

Tried crewAi with mistral:7b-instruct-v0.3. Still sucks...