API should allow defining GPU layers when loading a model

KoboldAI / KoboldAI-Client

For GGUF support, see KoboldCPP: https://github.com/LostRuins/koboldcpp

https://koboldai.com

GNU Affero General Public License v3.0

3.47k stars 747 forks source link

API should allow defining GPU layers when loading a model #317

Open Pyroglyph opened 1 year ago

Pyroglyph commented 1 year ago

At the moment the API does not allow you to set the number of GPU layers when you PUT a model. This forces all API-driven applications to run on the CPU, which seems like a bit of an oversight.

I've done some basic testing and got something working, so I'll send a PR over soon.