search

mudler / LocalAI

:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and local-first. Drop-in replacement for OpenAI, running on consumer-grade hardware. No GPU required. Runs gguf, transformers, diffusers and many more models architectures. Features: Generate Text, Audio, Video, Images, Voice Cloning, Distributed inference
https://localai.io
MIT License
23.3k stars 1.77k forks source link

support functionary models #1717

Open mudler opened 7 months ago

mudler commented 7 months ago

Functionary models are trained to output functions.

We should just provide a flag to turn off calculating BNF grammars as we are already doing and just leave to the LLM to answer appropriately.

bdqfork commented 6 months ago

It may also be necessary to provide a method to inject the definition of the tool as json string into the prompt template.

mudler commented 5 months ago

It may also be necessary to provide a method to inject the definition of the tool as json string into the prompt template.

that's already present, you can render in the template by ranging over {{.Functions}}. See for example: https://github.com/mudler/LocalAI/blob/f9c75d487851749d3b382f64bb3d8a9bf52d94dd/embedded/models/hermes-2-pro-mistral.yaml#L23

olariuromeo commented 2 weeks ago

Will be nice , to use functionary models do you have already a yaml recipe for the functionary models?