search

makcedward / nlpaug

Data augmentation for NLP
https://makcedward.github.io/
MIT License
4.46k stars 463 forks source link

The tokenizer class you load from this checkpoint is 'RobertaTokenizer' #335

Open bhomass opened 1 year ago

bhomass commented 1 year ago

I ran thru the lambada-train_model.ipynb example on my data, and got the tokenizer error 

The tokenizer class you load from this checkpoint is 'RobertaTokenizer'. 
The class this function is called from is 'GPT2Tokenizer'.

There is no mention of tokenizer in any of the call parameters. How did this tokenizer mix up occur?