CLUEbenchmark / CLUEPretrainedModels

高质量中文预训练模型集合:最先进大模型、最快小模型、相似度专门模型
https://arxiv.org/abs/2003.01355
803 stars 96 forks source link

transformers中使用clue/roberta_chinese_pair_tiny的疑问 #23

Open zy614582280 opened 2 years ago

zy614582280 commented 2 years ago

我通过transformers使用roberta_chinese_pair_tiny,提示以下warning

  1. You are using a model of type roberta to instantiate a model of type bert. This is not supported for all configurations of models and can yield errors.
  2. Some weights of the model checkpoint at clue/roberta_chinese_pair_tiny were not used when initializing BertModel: ['cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.predictions.decoder.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.weight']

我的代码如下:

from transformers import BertModel bert = BertModel.from_pretrained("clue/roberta_chinese_pair_tiny")

brightmart commented 2 years ago

warning是不是可以认为是只是提示啊。反正最后一层相关的参数,你是需要自己训练过的。