Model outputs are not passed to compute_metrics functions completely

[Johnny785]

System Info

python version 3.8.5, transformers version 4.19.1

Who can help?

No response

Information

• [ ] The official example scripts
• [X] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [X] My own task or dataset (give details below)

Reproduction

1. Create a model class based on a base model class in transformers
2. Overwrite its forward method to return the logits, the scores, and another output that is a tensor and has a 0 dimension higher than the batch size(such as 3 times the batch size)
3. Initialize a Trainer class to train the model as usual, pass in a custom compute_metrics function
4. In the custom compute_metrics function, print out the custom output as outlined above from EvalPrediction.predictions, and check its dimensions

Expected behavior

I customized my forward function to return an input field in the dictionary, which is of size 5BxC where B is the batch size. Other metrics such as loss and logits are BxC or B. However, in the compute_metrics function I customized, I received only BxC data entries from the EvalPrediction.predictions attribute for the input I returned in forward. Other fields that have the first dimension equal to B are passed in correctly. When I changed my input field shape to Bx5xC, it is also passed in correctly.

I read through the code that calls the compute_metrics function, however, I did not find a piece of code that trims the output returned from the model. After experimenting, I found that it's not taking mean, taking sum or taking max.

I would really appreciate an explanation as to why this happens, thank you for your assistance

[amyeroberts]

Hi @Johnny785, thanks for opening an issue!

So that we can best help you, could you:

• Provide more running environment information: run transformers-cli env in the terminal and copy-paste the output
• Provide a minimal code reproducer we can run to replicate the issue. We get many issues and PRs opened per day, so do not have the time to translate written descriptions into code. Making it as easy as possible for us to run and debug is the surest way of us being able to help.

As this involves custom code and overriding standard methods, this is a question best placed in our forums. We try to reserve the github issues for feature requests and bug reports.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.