search

vercel / modelfusion

The TypeScript library for building AI applications.
https://modelfusion.dev
MIT License
1.01k stars 76 forks source link

Add support for llama.cpp cache_prompt parameter #202

Closed djwhitt closed 7 months ago

djwhitt commented 7 months ago

llama.cpp recently added a cache_prompt parameter to its API: https://github.com/ggerganov/llama.cpp/commit/05cd6e5036d72d0930de4d8f6be7bce09e8dda24

It does the following (described in the llama.cpp server README):

cache_prompt: Save the prompt and generation for avoid reprocess entire prompt if a part of this isn't change (default: false)

This PR adds support for it. The build runs and the functionality seems to work as intended in my project.

vercel[bot] commented 7 months ago

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
modelfusion ✅ Ready (Inspect) Visit Preview 💬 Add feedback Dec 10, 2023 5:10pm
lgrammel commented 7 months ago

Neat, thank you!

lgrammel commented 7 months ago

Included in v0.95.0