Closed Skylixia closed 3 years ago
The same happens with the fine tuned model on Dutch books
Thank you for noticing this issue. We are aware of this and will release a fix later.
While we are fixing this, you can use fairseq for language masking. In this library, the masking works as expected.
As an example, we have a MaskedLMAdapter
that uses fairseq which you can use. You can find that here on this repo.
You can download the fairseq models here under the fairseq tab.
Thank you @iPieter ! I downloaded the fairseq model but I only get the model.pt file. To load a pretrained model with fairseq the vocabulary file is also needed (dict.txt) and bpecodes (see loading a custom model ). Is it downloadable separately somewhere else ? Thank you !
@Skylixia You can use the default RoBERTa ones:
wget -N 'https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/encoder.json'
wget -N 'https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/vocab.bpe'
wget -N 'https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/dict.txt'
Any update on this?
We recently released RobBERT v2 (see readme), which you can download the Fairseq model here. This uses a Dutch tokenizer.
On HuggingFace, while RobBERT-v2's base model is on there and working as it should, it is currently missing its MLM head due to a bug in the conversion from the fairseq model to the huggingface transformer. @iPieter is currently looking into this!
We updated the MLM head on the Huggingface library last week. It should now be fully compatible with the Fairseq model. In case you're interested in what changed: there was a bias term not correctly copied to the huggingface model.
The updated model has the same identifier (pdelobelle/robbert-v2-dutch-base
)
In addition, I also just added a notebook on the MLM head, that also uses pipelines.
Hello,
I've loaded RobBert with huggingface transformers and wanted to predict mask with it but I get surprising results. What am I doing wrong ?
I've also tried with the AutoModel class