LostRuins
commented
1 year ago Fyi Koboldcpp already supports sending any context size you want by manually editing the value inside the Lite UI (it's a text input)
However, looking at the llama.cpp code they are still using fixed sized scratch buffers for the context, and they are also fixing the memory allocation at load time.
Can you confirm that this model works correctly in llama.cpp at 4096 context? If it does, then it should work in KoboldCpp too. I will see if I can get the buffers resized correctly
Edit: I can confirm that they've added more hackery that only resizes the KV caches at load time (previously it was dynamically resized during eval. I will add a new parameter to set max ctx len)
ghost
JHawkley
This is a feature from llama.cpp that kobold.cpp currently does not support.
The recently released Bluemoon RP model was trained for 4K context sizes, however, in order to make use of this in llama.cpp, you must use the
--ctx_size 4096flag to enable larger contexts without seg-faulting.kobold.cpp has no support for this flag, and so this model cannot be utilized to its fullest potential. You can check this comment/discussion on HuggingFace going over this for more context.
Note: it does appear that a LLaMA model must be finetuned for 4k, otherwise you just get gibberish. Unfortunately, this will not automagically expand the context of all LLaMA models, but it's pretty cool that a finetune can add this capability. We may see more models tuned like this in the future.