[Feature]: DRY Sampling

Shreyansh1311 commented 1 month ago

🚀 The feature, motivation and pitch

DRY is a sampler that completely mitigates repetitions. This is especially important for small models which tend to slop in large contexts. Here's an explanation of DRY from the author himself https://github.com/oobabooga/text-generation-webui/pull/5677 Along with oobabooga, koboldcpp also has an implementation of DRY which according to author IIRC is better than oobabooga's DRY has been a completely game changer for me and from what I have seen several others. It completely removes need for other samplers like top_p, top_k, repetition_penalty. It is recommended to be used with min_p and produces great coherent results.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

AlpinDale commented 1 month ago

I've been attempting to implement this, but dry_sequence_breakers has been an absolute chore, and made me somewhat give up. If someone wants to collaborate on this, please let me know. I've done most of the work, just need one final push with some issues related to that.

Shreyansh1311 commented 1 month ago

Oh that's really wonderful @AlpinDale. I am afraid I am not smart enough to help, otherwise would definitely have volunteered. @p-e-w would you be the able to help with this. It will be really great to have DRY implemented for batch inferences as well.

p-e-w commented 1 month ago

I'm happy to review a pull request and answer any questions. However, I suggest waiting for a maintainer to comment on whether they actually want DRY in vLLM.

Shreyansh1311 commented 1 month ago

@p-e-w I am not sure if @AlpinDale was talking about vLLM or aphrodite-engine, but I guess it will be equally good to have it in aphrodite, if that's what AlpinDale meant.