Open Yandrik opened 1 month ago
Hmm ok so an EOS token is missing - I'll check this - thanks for the report
Last week, I updated unsloth to the latest version. Today I also encountered this problem.
Yep working on a fix!
Apologies - hopefully it's fixed now!
I encountered an issue when trying to export a GGUF model file for Mistral Nemo and Mistral 7B finetunes using the
unsloth
library. The error occurs during thesave_pretrained_gguf
function call, specifically when creating theollama_modelfile
. TheKeyError: '__EOS_TOKEN__'
is raised, which crashes the process. This problem happens with themistral
andllama
templates, but not withllama-3
orphi-3
.Steps to Reproduce
mistral
(it seems like bothmistral
andllama
templates result in the error).model.save_pretrained_gguf()
function. (we used quantization too, but it seems like the quant works fine, so that part is probably irrelevant)KeyError: '__EOS_TOKEN__'
during thecreate_ollama_modelfile
step.Workaround
I attempted to set
tokenizer._ollama_modelfile
toNone
as a workaround, but this approach doesn't actually work consistently.Environment
Additional Information
The issue does not occur with
llama-3
orphi-3
templates. The quantization to Q4_K_M GGUF completes successfully; the problem lies in saving the modelfile.