Closed vince62s closed 2 months ago
When using tokenizer.model (mostly older llama1/2 and mistral) with sentencepiece, look also at tokenizer.json and get the "added_tokens" which are not in the sentencepiece model vocab.
When using tokenizer.model (mostly older llama1/2 and mistral) with sentencepiece, look also at tokenizer.json and get the "added_tokens" which are not in the sentencepiece model vocab.