search

firebase / genkit

An open source framework for building AI-powered apps with familiar code-centric patterns. Genkit makes it easy to develop, integrate, and test AI features with observability and evaluations. Genkit works with various models and platforms.
Apache License 2.0
802 stars 121 forks source link

[Docs] which models supports tools? #661

Closed xster closed 3 months ago

xster commented 4 months ago

Trying out a few ollama models. But they fail with

Model "ollama/qwen2" does not support tools, but some tools were supplied to generate(). Please call generate() without tools if you would like to use this model.

Even phi3 or gemma2 doesn't support tools. Well which ones do support tools?

MichaelDoyle commented 4 months ago

Hey @xster. Currently ollama does not support tool (function) calling. Models that do support it include:

Tool (or function) calling is generally dependent on availability from the model provider. It's my understanding that ollama does not provide a tool calling API. I have seen there are other frameworks out there that provide "wrappers" that implement tool calling for any model so long as it can provide structured output (i.e. json).

This is something we could discuss and consider either internally (perhaps as a plugin?) or as a community plugin.

xster commented 4 months ago

Not supporting ollama is ok. It should just be documented. Either here https://firebase.google.com/docs/genkit#4_give_llms_tools or in a breakout page about tool use (though I notice tool doesn't have a page)

xster commented 4 months ago

Though it seems like langchain somehow supports tools via ollama

MichaelDoyle commented 4 months ago

I see. Yes, the docs for tool calling can definitely be improved -- we'll prioritize that. I agree, on the page you linked to, things are worded in such a way that it appears that "tool calling" is a Genkit provided feature, and not dependent on model specific capabilities. The Generate Content page does mention tool calling is a model specific feature: https://firebase.google.com/docs/genkit/models#tools_and_function_calling, but its not very helpful.

I am not sure it would be prudent for us to list model by model what features are supported; model specific documentation is the source of truth for that. But maybe there is a way for us to make "supported features" more easily discoverable in the framework. You can see this pretty easily in the code if you know where to look. Here is ollama: https://github.com/firebase/genkit/blob/main/js/plugins/ollama/src/index.ts#L72-L75

Wrt langchain, yes, they are doing this via an abstraction/wrapper layer. We could consider the same, perhaps via middleware. I'll open a FR for this.

xster commented 4 months ago

Also see https://ollama.com/blog/tool-support

MichaelDoyle commented 3 months ago

Fixed via https://github.com/firebase/genkit/pull/809