我参考了一下hugging face的无监督训练代码,简单测试了一下。代码如下:
from transformers import T5Tokenizer, T5ForConditionalGeneration
import torch
tokenizer = T5Tokenizer.from_pretrained("premodel/ChatYuan-large-v1")
model = T5ForConditionalGeneration.from_pretrained("premodel/ChatYuan-large-v1")
training
input_ids = tokenizer("一只走在大街上", return_tensors="pt").input_ids
labels = tokenizer("可爱的宽敞的", return_tensors="pt").input_ids
outputs = model(input_ids=input_ids, labels=labels)
loss = outputs.loss
logits = outputs.logits
结果报错:
IndexError: index out of range in self
应该是embedding层索引越界,看了模型的词表,并没有标记,但是tokenizer后没有报错
请问如何基于YUAN模型进行无监督预训练?无监督预训练的数据格式是什么,万分感谢
我参考了一下hugging face的无监督训练代码,简单测试了一下。代码如下: from transformers import T5Tokenizer, T5ForConditionalGeneration import torch tokenizer = T5Tokenizer.from_pretrained("premodel/ChatYuan-large-v1") model = T5ForConditionalGeneration.from_pretrained("premodel/ChatYuan-large-v1")
training
input_ids = tokenizer("一只走在大街上", return_tensors="pt").input_ids
labels = tokenizer("可爱的宽敞的", return_tensors="pt").input_ids
outputs = model(input_ids=input_ids, labels=labels)
loss = outputs.loss
logits = outputs.logits
结果报错:
IndexError: index out of range in self
应该是embedding层索引越界,看了模型的词表,并没有标记,但是tokenizer后没有报错
请问如何基于YUAN模型进行无监督预训练?无监督预训练的数据格式是什么,万分感谢