Feature: Tail Free Sampling

[mrseeker]

🚀 Feature request

I would like to have Huggingface implement Tail Free Sampling, or TFS for short, to the official repo. The original paper for this can be found here: https://trentbrick.github.io/Tail-Free-Sampling/#tail-free-sampling-algorithm

Implementation of this code can be found here: https://github.com/finetuneanon/transformers/blob/gpt-neo-localattention3-rp-b/src/transformers/generation_logits_process.py#L243-L284

Motivation

At the moment KoboldAI uses Finetuneanon's version of Transformers for creating stories, however, this version has several changes made to the repository that make it impossible for me to use (eg: I use an aarch64 architecture with an integrated Nvidia chip, which is only supported by Huggingface's main branch). There are calls from the KoboldAI devs to migrate due to the integrated support of GPT-J in Huggingface, however, there are some features in Finetuneanon's Transformer which are not found in the main branch. Tail Free Sampling is one such feature.

Your contribution

I would like to file a PR with the code, however, I do lack the knowledge on how to implement the required tests and checks. An example is available on Finetuneanon's branch but it will require cherry-picking the commit to the main branch.

[patil-suraj]

cc @patrickvonplaten

[patrickvonplaten]

@mrseeker - would you like to open a PR for it :-) ?

[mrseeker]

Bumping to prevent getting closed, PR is WIP.