search

Maximilian-Winter / llama-cpp-agent

The llama-cpp-agent framework is a tool designed for easy interaction with Large Language Models (LLMs). Allowing users to chat with LLM models, execute structured function calls and get structured output. Works also with models not fine-tuned to JSON output and function calls.
Other
472 stars 42 forks source link

How to make it work with llama.cpp OpenAI-like server? #4

Closed tranhoangnguyen03 closed 8 months ago

tranhoangnguyen03 commented 9 months ago

For my usecase, I host the llama.cpp at a remote server to serve my frontend codes. Does llama-cpp-agent work with this setup?

Maximilian-Winter commented 9 months ago

No, not at the moment, but I will add llama.cpp server support next.

Maximilian-Winter commented 8 months ago

@tranhoangnguyen03 I added support for the llama.cpp backend server and OpenAI endpoints that have grammar support like the llama-cpp-python server.