LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.98k stars 349 forks source link

gemma2: add sliding window mask #959

Closed Nexesenex closed 3 months ago

Nexesenex commented 3 months ago

Expand the usable context from 5k to 8k.

https://github.com/ggerganov/llama.cpp/pull/8227

Additional commits :

fix data_swa uninitialized better naming add co-author

Nexesenex commented 3 months ago

The PR works as it is beyond the previous 5k context cap for Gemma V2 softcap. Slaren just wants to simplify the code.

LostRuins commented 3 months ago

I'll think i will wait to merge this when it's merged in llama.cpp

Nexesenex commented 3 months ago

And merged it is, with a further drop in perplexity ! :D