search

makcedward / nlpaug

Data augmentation for NLP
https://makcedward.github.io/
MIT License
4.41k stars 460 forks source link

The tokenizer class you load from this checkpoint is 'RobertaTokenizer' #335

Open bhomass opened 1 year ago

bhomass commented 1 year ago

I ran thru the lambada-train_model.ipynb example on my data, and got the tokenizer error 

The tokenizer class you load from this checkpoint is 'RobertaTokenizer'. 
The class this function is called from is 'GPT2Tokenizer'.

There is no mention of tokenizer in any of the call parameters. How did this tokenizer mix up occur?