LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
5.33k stars 363 forks source link

Dynamic sizes for sequences #1157

Closed mayaeary closed 1 month ago

mayaeary commented 1 month ago

Make dry sequences, stop sequences, logit biases and banned strings to be allocated dynamically, to get rid of hard limit.

LostRuins commented 1 month ago

Hm, I don't think this is a good idea. Feels like it'll be prone to either memory leaks when the container gets resized, or going out of scope prematurely. Especially since I don't have control of when gc happens

LostRuins commented 1 month ago

Will review this again when I have some time to test - hang tight for now

stepfunction83 commented 1 month ago

Could it be possible to specify this as a launch parameter to avoid any dynamic sizing issues?

LostRuins commented 1 month ago

Hi @mayaeary made some small tweaks, think this should be good to go.