Different results obtained using pipeline (worse) vs. model.generate under the same decoding strategy

[kirk86]

System Info

4.44 Linux 3.12

Who can help?

@Rocketknight1 @gante

Information

• [X] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [X] My own task or dataset (give details below)

Reproduction

1. Train a small model like T5 on a small synthetic data for summarization
2. Evaluate the model on the test set
3. Results match validation accuracy from train logs when using model.generate with default decoding_strategy = greedy search when setting do_sample=False, num_beams=1
4. Try to replicate the same using the pipeline but results are completely off $\pm 50\%$
5. Double checked all parameters to be set to default values as in model.generate still the same issue.
6. Iterated through all possible decoding_strategies and still no sign of correction when using pipeline

Expected behavior

Expected the behaviour of pipeline to be the same as that of model.generate when using the same underlying decoding_strategy

[Rocketknight1]

Hi @kirk86 - could you upload your model (or reproduce the issue with an existing T5 model on the Hub), then share a short but complete reproducer script that shows the discrepancy? It'll make it a lot easier for us to track down any bugs!