ValueError: You have to specify either decoder_inputs or decoder_inputs_embeds

[puraminy]

Why there is no training example for T5 or MT5??? Could you please give me a link to an example? I had a hard time to write a code with various errors: This is my code:

import torch
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
#from transformers import MT5Model, T5Tokenizer
from transformers import MT5ForConditionalGeneration, T5Tokenizer

#tokenizer = AutoTokenizer.from_pretrained("google/mt5-base")
tokenizer = T5Tokenizer.from_pretrained("google/mt5-small")
raw_datasets = load_dataset("atomic")

def tokenize_function(examples):
    return tokenizer(examples["event"],max_length=128, padding="max_length", truncation=True)

def tokenize_labels(examples):
    with tokenizer.as_target_tokenizer():
         return tokenizer(examples["oReact"], return_tensors="pt")

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
#labels = raw_datasets.map(tokenize_labels, batched=True)
small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(1000))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(1000))
full_train_dataset = tokenized_datasets["train"]
full_eval_dataset = tokenized_datasets["test"]

from transformers import TrainingArguments

training_args = TrainingArguments("test_trainer")
#model = AutoModelForSeq2SeqLM.from_pretrained("google/mt5-base")
model = MT5ForConditionalGeneration.from_pretrained("google/mt5-small", torchscript = True)
#traced_model = torch.jit.trace(model, (input_ids, attention_mask, decoder_input_ids))

from transformers import Trainer

trainer = Trainer(
   model=model, args=training_args, train_dataset=small_train_dataset, eval_dataset=small_eval_dataset
)

trainer.train()
import numpy as np
from datasets import load_metric

metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(
    model=model,
   args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
)
trainer.evaluate()

I don't know how to feed the labels to this model...

And this is error:

...
 tr_loss += self.training_step(model, inputs)
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/transformers/trainer.py", line 1250, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/transformers/trainer.py", line 1277, in compute_loss
    outputs = model(**inputs)
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/transformers/models/t5/modeling_t5.py", line 1510, in forward
    return_dict=return_dict,
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/pouramini/miniconda3/lib/python3.7/site-packages/transformers/models/t5/modeling_t5.py", line 871, in forward
    raise ValueError(f"You have to specify either {err_msg_prefix}inputs or {err_msg_prefix}inputs_embeds")

@sgugger @patrickvonplaten @patil-suraj

[patil-suraj]

Hi there,

the summarization and translation examples supports fine-tuning T5 and mT5 (and other seq2seq models in the lib). Please take a look at the readme and the script.

The scripts are easily modifiable to support training on any seq2seq task.

Also there are multiple notebook on T5 training in community notebooks section. Hope that helps.

[puraminy]

Thank you very much! I haven't found the first examples which are up to date too.

I found community notebooks later, some of them are old. Maybe a recent one in the main notebooks is a good idea.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.