endless generation with this local model

[Wuzzooy]

Describe the bug

Hello, I found this model which i thought would be good for this task since it's a mixtral finetuned with function calling but i get endless non sens generation until the context set is reached(8192), i just say "hello". I'm using oobabooga and exllama2 format 5bit. https://huggingface.co/meetkai/functionary-medium-v2.2 In textgen webui i don't get this behavior. I don't have this issue with the original mixtral instruct and oobabooga as the backend. Does open interpreter uses its own sampling preset and prompt format or it's still the backend settings ? Thank you.

Reproduce

use this model https://huggingface.co/LoneStriker/functionary-medium-v2.2-5.0bpw-h6-exl2 and just start the chat.

Expected behavior

a short answer saying hello, how can i assist you today ?

Screenshots

No response

Open Interpreter version

0.2.0

Python version

3.11

Operating System name and version

windows 10

Additional context

No response

[Manamama]

A tip: you likely need to use the reply template. See e.g. the help here. The model tries to continue (not reply) then instead.

[jjv5]

A tip: you likely need to use the reply template. See e.g. the help here. The model tries to continue (not reply) then instead.

Help where? I have the same issue with all local models I've tried. Using version: Open Interpreter 0.2.0 New Computer

Endless loop of replies and going over the same thing. Not working at all even for the simplest task/query.

Is there an example of any local LLM that works with open-interpreter 0.2.0?

[Wuzzooy]

A tip: you likely need to use the reply template. See e.g. the help here. The model tries to continue (not reply) then instead.

Help where? I have the same issue with all local models I've tried. Using version: Open Interpreter 0.2.0 New Computer

Endless loop of replies and going over the same thing. Not working at all even for the simplest task/query.

Is there an example of any local LLM that works with open-interpreter 0.2.0?

I didn't understand the help thing neither but to answer to your question, yes i have some success with model like bagelmisterytour using oobabooga backend. Clearly not perfect but the model doesn't go crazy like the one i'm talking about on this topic and manage to perform some tasks. I still don't know what is handling the prompt format/sampling preset.

[Wuzzooy]

I've tried with llama.cpp python api server as the backend and i don't have the behavior i reported so i will close the issue.