Phi-3 mini 4k instruct tokenizer uses wrong EOS token

[DePasqualeOrg]

Using this model:

Prompt:

<s><|user|>
You are a helpful assistant. Name a number.<|end|>
<|assistant|>

Output:

123<|end|><|assistant|> 857<|end|><|assistant|> 567<|end|><|assistant|> 789<|end|><|assistant|> 456<|end|>

It looks like the EOS token for this model is set to <|endoftext|>, but it should be <|end|>. How can I override the default EOS token?

[DePasqualeOrg]

I can get this working with the Python MLX API by modifying the EOS token, but it looks like there isn't currently a way to do this with swift-transformers:

from mlx_lm import load, generate

tokenizer_config = {
  'eos_token': "<|end|>"
}

model, tokenizer = load("mlx-community/Phi-3-mini-4k-instruct-4bit", tokenizer_config=tokenizer_config)
response = generate(model, tokenizer, prompt="<s><|user|>\nName a color.<|end|>\n<|assistant|>\n", temp=0.5)
print(response)

Output:

Blue is a color.

[DePasqualeOrg]

https://github.com/ml-explore/mlx-swift-examples/issues/74