search

LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
5.02k stars 350 forks source link

Mistral Gen 2 template #1133

Open Vladonai opened 2 weeks ago

Vladonai commented 2 weeks ago

ver. 1.75.2

User tag:</s>\n[INST]
Assistant tag:[/INST]\n

Maybe the right thing to do User tag:</s>\n<s>\n[INST]

Then the system prompt will have the form

<s>
[INST]User's Instructions[/INST]
Assistant's response
</s>

That would make sense, I think.

LostRuins commented 2 weeks ago

You'll have to check with @pandora-s-git who is from Mistral and contributed the new templates

pandora-s-git commented 2 weeks ago

Hi there, It is important to not include the newlines, if any question regarding the templates please take a look at the following document explaining all the details: https://github.com/mistralai/cookbook/blob/main/concept-deep-dive/tokenization/chat_templates.md

All these white spaces are of extreme importance if we want it to match perfectly with the expected tokens.