Closed TrajansRow closed 1 year ago
This should be fixed in the latest version, currently v1.3 at time of writing.
In fact, I have made mmap disabled by default. To enable it, use the flag --usemmap
https://github.com/LostRuins/koboldcpp/releases/latest
Edit: for version 1.4 it is now mmap by default, you can toggle it off with --nommap
instead.
Thanks for the update! I'm seeing much better memory utilization now, although not quite the same sort of performance improvement I saw in llama.cpp (maybe a generation speedup on the order of 10ms/token, running 30B/4-bit llama on an M1 Max).
This is a great option to have in koboldcpp, but I don't think it should be enabled by default. For the majority of users, I don't expect the memory tradeoffs to provide meaningful benefit.
Yeah it was quite a divided response, some people hate it others love it. In the end it is a toggle so everyone can just pick whichever option they prefer.
There was a performance regression in earlier versions of llama.cpp that I may be hitting with long running interactions. This was recently fixed with the addition of a --no-mmap option which forces the entire model to be loaded into ram, and I would like to also use it with koboldcpp.
https://github.com/ggerganov/llama.cpp/pull/801