search

dranger003 / llama.cpp-dotnet

Minimal C# bindings for llama.cpp + .NET core library with API host/client.
MIT License
64 stars 7 forks source link

Issue with running some mistral models #24

Open HarvieKrumpet opened 1 week ago

HarvieKrumpet commented 1 week ago

Seen this happen on a few of the newer models. It loads ok, but upon tokenization getting a crash in...

llm.Tokenize(llmMessages).Length; Non-negative parameter is required (count)

Mistral-Nemo-Instruct-12B-Philosophy-Math.i1-Q6_K.gguf

Have not seen the tokenizer ever crash before, I can't see anything I am feeding it that would cause this either...

dranger003 commented 1 week ago

Did you use the one from here? https://huggingface.co/mradermacher/Mistral-Nemo-Instruct-12B-Philosophy-Math-i1-GGUF/tree/main

dranger003 commented 1 week ago

Looks like this is possibly an issue upstream, take a look at this: https://github.com/ggerganov/llama.cpp/issues/9864

You can apply this patch to fix the issue I think: https://github.com/ggerganov/llama.cpp/pull/9869/files