For making "to_hf_weights.py" work correctly, do I have to modify the following if I have my own tokenizer trained with vocab_size=50400? Or, can I assume that "GPT2Tokenizer" does not matter here as long as I use my own tokenizer to decode the inference result? (I expected so but my experiments show negative results.)
For making "to_hf_weights.py" work correctly, do I have to modify the following if I have my own tokenizer trained with vocab_size=50400? Or, can I assume that "GPT2Tokenizer" does not matter here as long as I use my own tokenizer to decode the inference result? (I expected so but my experiments show negative results.)
Thank you for your kind advice.