search

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
https://huggingface.co/transformers
Apache License 2.0
132k stars 26.29k forks source link

MLM training for DeBERTa not supported: configuration class is missing #9689

Closed xiaolin-cheng closed 3 years ago

xiaolin-cheng commented 3 years ago

When I ran the example script run_mlm.py to fine tune the pretrained deberta model on a customized dataset, I got the following error. The same command worked for roberta-base.

The command: python run_mlm.py --model_name_or_path 'microsoft/deberta-base' --train_file slogans/train.txt --validation_file slogans/test.txt --do_train --do_eval --per_device_train_batch_size 64 --per_device_eval_batch_size 64 --learning_rate 1e-3 --num_train_epochs 10 --output_dir /home/jovyan/share2/xiaolin/models/mlm/temp --save_steps 5000 --logging_steps 100

The terminal error: Traceback (most recent call last): File "run_mlm.py", line 409, in main() File "run_mlm.py", line 264, in main cache_dir=model_args.cache_dir, File "/home/jovyan/.local/lib/python3.7/site-packages/transformers/models/auto/modeling_auto.py", line 1093, in from_pretrained config.class, cls.name, ", ".join(c.name for c in MODEL_FOR_MASKED_LM_MAPPING.keys()) ValueError: Unrecognized configuration class <class 'transformers.models.deberta.configuration_deberta.DebertaConfig'> for this kind of AutoModel: AutoModelForMaskedLM. Model type should be one of LayoutLMConfig, DistilBertConfig, AlbertConfig, BartConfig, CamembertConfig, XLMRobertaConfig, LongformerConfig, RobertaConfig, SqueezeBertConfig, BertConfig, MobileBertConfig, FlaubertConfig, XLMConfig, ElectraConfig, ReformerConfig, FunnelConfig, MPNetConfig, TapasConfig.

NielsRogge commented 3 years ago

Looking at the docs, it seems like there's currently no DeBERTaForMaskedLM defined. I will make a PR that adds this.