Closed attardi closed 4 years ago
Allow using any models from Huggingface.
Some model's tokenizers do not provide a vocab field, so I create it as a dictionary from tokeniser._convert_id_to_token().
Great, thanks for your contribution.
Allow using any models from Huggingface.
Some model's tokenizers do not provide a vocab field, so I create it as a dictionary from tokeniser._convert_id_to_token().