search

LostRuins / koboldcpp

A simple one-file way to run various GGML and GGUF models with KoboldAI's UI
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.35k stars 312 forks source link

Benchmark context size limited to 16284 #878

Open Barafu opened 1 month ago

Barafu commented 1 month ago

When using benchmark feature, the actual size of text context is limited by 16284 when the --contextsize is set to 16384 or greater. Can it be made to utilise full allowed context for more reliable results?

LostRuins commented 1 month ago

Alright, but for very extreme cases where the context size is very large, the benchmark prompt may not be long enough to trigger a max context generation.

Barafu commented 1 month ago

Thanks. I haven't read the benchmark prompt, but you can probably simply duplicate its definition part. At least for me, duplicating the prompt has little effect on the output.

LostRuins commented 1 month ago

This is now unlocked in 1.67 release