Diverse beam search

[joecummings]

Feature Description

Add ability for diverse beam search according to this paper: https://ojs.aaai.org/index.php/AAAI/article/view/12340

Use Case

"DBS produces sequences that are significantly different from each other by incorporating diversity constraints within groups of candidate sequences during decoding; moreover, it achieves this with minimal computational or memory overhead."

Additional Context

Request from TorchText partners.

• HuggingFace API: https://huggingface.co/docs/transformers/v4.26.1/en/main_classes/text_generation#transformers.GenerationMixin.group_beam_search

[joecummings]

@jacobkahn I'm planning on taking a stab at this.

[jacobkahn]

We'll track the PR here.