search

philschmid / deep-learning-pytorch-huggingface

MIT License
618 stars 143 forks source link

compute_metrics() function #3

Open ybagoury opened 1 year ago

ybagoury commented 1 year ago

The compute_metrics function takes as a parameter eval_preds which is unpacked into preds and labels. Here's the line mentioned:

preds, labels = eval_preds

What I'm trying to figure out is what exactly eval_preds is . When it is called in the Trainer, it doesn't contain any arguments.

trainer = Seq2SeqTrainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["test"],
    compute_metrics=compute_metrics,
)

If my understanding is correct it's a dictionary that contains the labels we already have in tokenized_dataset["train"]["labels"]and the predictions of labels that are going to be compared to using the rouge score. So where are these labels coming from?

Thanks.

sadransh commented 1 year ago

eval_preds is an object of type EvalPredictions. This is what's happening: https://github.com/huggingface/transformers/blob/080a97119c0dabfd0fb5c3e26a872ad2958e4f77/src/transformers/trainer.py#L3237C1-L3243C3

Also I belive such questions are better to be asked on hugging face forum as it is related to transformers package, so the whole community will benefit from discussion :)