Trainer do model generation during evaluation loop

huggingface / transformers

🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

https://huggingface.co/transformers

Apache License 2.0

132.39k stars 26.36k forks source link

Trainer do model generation during evaluation loop #23763

Closed szxiangjn closed 1 year ago

szxiangjn commented 1 year ago

Feature request

Current trainer only supports teacher-forcing generation for computing evaluation loss but not auto-regressive generation for other metrics. Seq2SeqTrainer supports this but seems that it only accepts encoder-decoder models like T5 instead of GPT-style (decoder-only) models. Would this feature be added in the future?

Motivation

I am training a decoder-only model and want to use model.generate to evaluate it during training.

Your contribution

I haven't investigated deeply into Trainer code.

sgugger commented 1 year ago

You can write your own subclass of the Trainer, it's not supported and we don't plan on adding it.

github-actions[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

JackCai1206 commented 3 months ago

Why should this not be supported? I think it makes sense to have a predict_with_generate option.