[BUG] mlx_lm issue with Phi-3 fine tuned model: adding and repeating weird tokens

[IzzyHibbert]

Describe the bug Using mlx_lm I made a fine tuned model from microsoft/Phi-3-mini-128k-instruct. Training was all fine. AFter it, when I test the tuned model with generate I noticed one issue in the format of the prompt reply, which comes with multiple repetition of a <unk> like :

Have you considered discussing this with your friends or family?<unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk><unk>

To Reproduce

My code snippet

response = generate(
    fused_model,
    fused_tokenizer,
    prompt=format_prompt(system_prompt, question),
    verbose=True,  
    temp=0.0,
    max_tokens=512,
)

The system_prompt I am using is the specific one coming from microsoft/Phi-3-mini-128k-instruct, so it's like :

<|system|>
## Instruction
You are a helpful AI assistant.<|end|>
<|user|>
## User
Where is Paris ?<|end|>
<|assistant|>

The dataset used is vibhorag101/phr-mental-therapy-dataset-conversational-format-1024-tokens

Expected behavior I made the same fine-tuning for other models, and using the same dataset. I haven't found the "additional tokens" issue.. The expectation is to have the answer only (no other repetition of tokens)

Desktop (please complete the following information):

• OS Version: MacOS 14.1
• mlx-lm: latest

[mzbac]

Interesting, I am not aware of Phi3 supporting "sysmte" in the prompt. It looks like there is something wrong with the dataset preprocessing. I have done some Phi3 fine-tuning and it works fine for me. FYI https://huggingface.co/mzbac/Phi-3-mini-4k-instruct-function-calling

[IzzyHibbert]

Interesting, I am not aware of Phi3 supporting "sysmte" in the prompt. It looks like there is something wrong with the dataset preprocessing. I have done some Phi3 fine-tuning and it works fine for me. FYI https://huggingface.co/mzbac/Phi-3-mini-4k-instruct-function-calling

Good point. Time to retry...