search

PaddlePaddle / PaddleNLP

👑 Easy-to-use and powerful NLP and LLM library with 🤗 Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including 🗂Text Classification, 🔍 Neural Search, ❓ Question Answering, ℹ️ Information Extraction, 📄 Document Intelligence, 💌 Sentiment Analysis etc.
https://paddlenlp.readthedocs.io
Apache License 2.0
11.99k stars 2.93k forks source link

[Question]: 没有找到载入XLMRobertaModel的方法 #5981

Closed 5i-wanna-be-the-666 closed 4 months ago

5i-wanna-be-the-666 commented 1 year ago

请提出你的问题

我查找了api文档，里面只有载入XLM的方法，请问是现在暂时没有实现XLMRobertaModel模型吗？

如果是和XLM库融合在一起了，那么我在蒸馏学习的时候是否能使用这样的代码进行实现呢

from paddlenlp.transformers import XLMModel, XLMTokenizer

teacher_tokenizer = XLMTokenizer.from_pretrained("path/to/my_model")
teacher_model = XLMModel.from_pretrained("path/to/my_model")

student_tokenizer = XLMTokenizer.from_pretrained("xlm-mlm-tlm-xnli15-1024")
student_model = XLMModel.from_pretrained("xlm-mlm-tlm-xnli15-1024")

如果上面的方法不可行的话，不知道在general_distill.py中，能否通过下面的方法引用xlm-r模型呢？

from transformers import XLMRobertaModel, XLMRobertaTokenizer
MODEL_CLASSES = {
    "tinybert": (TinyBertForPretraining, TinyBertTokenizer),
    "bert": (BertForSequenceClassification, BertTokenizer),
    "xlm-r":(XLMRobertaModel,XLMRobertaTokenizer),
}
JunnYu commented 1 year ago

你好，当前没有实现XLMRoberta模型