search

kingoflolz / mesh-transformer-jax

Model parallel transformers in JAX and Haiku
Apache License 2.0
6.27k stars 890 forks source link

save_config_to_hf_format() #190

Open leejason opened 2 years ago

leejason commented 2 years ago

For making "to_hf_weights.py" work correctly, do I have to modify the following if I have my own tokenizer trained with vocab_size=50400? Or, can I assume that "GPT2Tokenizer" does not matter here as long as I use my own tokenizer to decode the inference result? (I expected so but my experiments show negative results.) 

def save_config_to_hf_format(......):

config = {
  ......
  "tokenizer_class": "GPT2Tokenizer"
  ......
}

Thank you for your kind advice.