Closed eevers-avisi closed 2 years ago
Thanks for your interest in RobBERT. That error was caused by an upgrade of the Tokenizer library, which changed the way the vocabulary gets loaded in. Fixing this is a bit of a hassle, which is why it took so long...
You can find the updated model here: https://huggingface.co/pdelobelle/robbert-v2-dutch-ner
I hope it's still useful for you!
Hi @iPieter,
I was trying to use your robbert-v2-dutch-ner model in my code for fine-tuning. I would like to use the tokenizer as a fast tokenizer, so that I'm able to use the word id's to know from which words the tokens do origin.
Unfortunately I'm not able to create a RobertaTokenizerFast from the pretrained model because of the following error: Error while initializing BPE: Token
Ċ
out of vocabularyWhen trying to find a solution, I saw the following issue (https://github.com/huggingface/transformers/issues/9290) which mentions almost the same problem (I think) for the robbert-v2-dutch-base model.
Is it possible that the same fix applied for the base model is also applied for the NER model?