search

henk717 / KoboldAI

KoboldAI is generative AI software optimized for fictional use, but capable of much more!
http://koboldai.com
GNU Affero General Public License v3.0
359 stars 130 forks source link

Add KoboldCpp library as GGML backend for KoboldAI #436

Closed LostRuins closed 1 year ago

LostRuins commented 1 year ago

Please test and review before merging. This is very much a proof-of-concept.

This PR adds KoboldCpp as a modular GGML backend to KoboldAI. It does not use the API, instead interfaces with KoboldCpp.py's python bindings to directly invoke cpp functions. Bundles KoboldCpp as prebuilt windows dlls. This is the first MVP, without additional customizability or accelerators included.

To use, this is an example launch command:

play.bat --model_backend ggml --model C:\path_to_models\airoboros-7b-gpt4.ggmlv3.q4_0.bin

The model name expects the text ggml inside the filename.

LostRuins commented 1 year ago

converting to draft PR pending further review.

LostRuins commented 1 year ago

Added a basic linux binary, which should be automatically selected on linux based OS.

henk717 commented 1 year ago

For legal reasons and self compilability the PR should bundle koboldcpp's source code.

LostRuins commented 1 year ago

agreed, let me rework this such that it can function as a submodule pulled from the koboldcpp repo