Feature Request: --multiuser they will be executed in sequence. Koboldcpp does not allow parallel decoding.Multi task parallel processing, hoping next update

LostRuins / koboldcpp

A simple one-file way to run various GGML and GGUF models with KoboldAI's UI

https://github.com/lostruins/koboldcpp

GNU Affero General Public License v3.0

4.35k stars 312 forks source link

Feature Request: --multiuser they will be executed in sequence. Koboldcpp does not allow parallel decoding.Multi task parallel processing, hoping next update #870

Closed windkwbs closed 1 month ago

windkwbs commented 1 month ago

Feature Request: --multiuser they will be executed in sequence. Koboldcpp does not allow parallel decoding.Multi task parallel processing, hoping next update