Any easy way to use Ollama endpoints?

Maximilian-Winter / llama-cpp-agent

The llama-cpp-agent framework is a tool designed for easy interaction with Large Language Models (LLMs). Allowing users to chat with LLM models, execute structured function calls and get structured output. Works also with models not fine-tuned to JSON output and function calls.

Other

445 stars 38 forks source link

Any easy way to use Ollama endpoints? #32

Open sesa507703 opened 3 months ago

sesa507703 commented 3 months ago

Hello. Our VMs are already supporting Ollama server and I want to reuse with this projects. I tried a few things but no luck. Any suggestions will be appreciated.

model = OpenAIEndpointSettings(completions_endpoint_url="http://localhost:11434/v1/chat/completions") No luck here.

Maximilian-Winter commented 3 months ago

Ollama doesn't support grammar guided generation as far as I know. So my framework won't work.

Maximilian-Winter commented 2 months ago

@sesa507703 I think Ollama merged support for JSON schema, I'm currently adding support for this.