When loading from huggingface, the tokenizer for Qwen has length 151851 (using the AutoTokenizer class as is done in the training script).
However, the final length of the tokenizer of your model is 151860. In the training should we be initializing the custom QwenTokenizer class instead and loading the qwen.tiktoken file or do we need to manually add the 9 special tokens?
When loading from huggingface, the tokenizer for Qwen has length 151851 (using the AutoTokenizer class as is done in the training script).
However, the final length of the tokenizer of your model is 151860. In the training should we be initializing the custom QwenTokenizer class instead and loading the
qwen.tiktokenfile or do we need to manually add the 9 special tokens?