Open ai-made-approachable opened 5 months ago
I think there is something to do with the HF tokenizer behavior. I can see that <sep>
is in https://huggingface.co/berkeley-nest/Starling-LM-7B-alpha/blob/main/tokenizer_config.json#L55. but it doesn't exist in https://huggingface.co/berkeley-nest/Starling-LM-7B-alpha/blob/main/tokenizer.json. somehow it has been added as new special token.
When I run mlx_lm.convert for berkeley-nest/Starling-LM-7B-alpha my mlx model suddenly has 32003 instead of 32002 tokens. This creates issues if you want to train and later export a .gguf file via llama.cpp
Original models added_token.json (https://huggingface.co/berkeley-nest/Starling-LM-7B-alpha/blob/main/added_tokens.json)
added_token.json after converting it to mlx