search

LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.66k stars 334 forks source link

Use this as an api from another python file? #18

Closed regstuff closed 1 year ago

regstuff commented 1 year ago

Hi, First off, thanks for the OPENBLAS tip. That cuts down the initial prompt processing time by like 3-4x! Was wondering if its possible to use the generate function as an api from another python file. Secondly, is it possible to update to the latest llama.cpp with a git pull from the llama.cpp library or do I have to wait for you to sync changes and then git pull from koboldcpp.

LostRuins commented 1 year ago

Yes, the API is a KoboldAI compatible API and can be used by any external program. Take a look at the attached file, the main endpoint that you want to use is /api/v1/generate

The repo is manually synced with the parent one every few days, but you'd probably want to wait for me to pull updates manually as there are often breaking incompatibilities that arise after merging which I have to fix before making a new version. It's not exactly plug and play due to the multiple changes I've made.

koboldapi-documentation.pdf