search

LostRuins / koboldcpp

A simple one-file way to run various GGML and GGUF models with KoboldAI's UI
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.35k stars 312 forks source link

Feature request: auto sleep #892

Open baronvertigovongrahamthesecondofsealand opened 4 weeks ago

baronvertigovongrahamthesecondofsealand commented 4 weeks ago

Is it at all possible to have the model go to sleep (ie. not use any resources) if conversation hasnt happened in a certain timeout value?

LostRuins commented 4 weeks ago

KoboldCpp is idle the moment the generation is complete - it does not use any CPU resources when not generating. The memory will stay in use until the model is unloaded - this will not happen automatically. You'd need some external script to start and close kobold as needed.