search

KoboldAI / KoboldAI-Client

For GGUF support, see KoboldCPP: https://github.com/LostRuins/koboldcpp
https://koboldai.com
GNU Affero General Public License v3.0
3.47k stars 747 forks source link

Allow setting GPU layers through the API #318

Closed Pyroglyph closed 1 year ago

Pyroglyph commented 1 year ago

This PR enables setting GPU layers through the API by passing a breakmodel layer string alongside the model name.

I'm not a Python developer so if something looks odd that's why. However, I've done some testing locally and it seems to work fine without any errors.

henk717 commented 1 year ago

Please submit this to our dev branch at https://github.com/henk717/koboldai

Pyroglyph commented 1 year ago

Done. New replacement PR is here.