[Feature]: DRY Sampling

vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs

https://docs.vllm.ai

Apache License 2.0

31.03k stars 4.72k forks source link

[Feature]: DRY Sampling #8581

Open Shreyansh1311 opened 2 months ago

Shreyansh1311 commented 2 months ago

🚀 The feature, motivation and pitch

DRY is a sampler that completely mitigates repetitions. This is especially important for small models which tend to slop in large contexts. Here's an explanation of DRY from the author himself https://github.com/oobabooga/text-generation-webui/pull/5677 Along with oobabooga, koboldcpp also has an implementation of DRY which according to author IIRC is better than oobabooga's DRY has been a completely game changer for me and from what I have seen several others. It completely removes need for other samplers like top_p, top_k, repetition_penalty. It is recommended to be used with min_p and produces great coherent results.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

[X] Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

AlpinDale commented 2 months ago

I've been attempting to implement this, but dry_sequence_breakers has been an absolute chore, and made me somewhat give up. If someone wants to collaborate on this, please let me know. I've done most of the work, just need one final push with some issues related to that.

Shreyansh1311 commented 2 months ago

Oh that's really wonderful @AlpinDale. I am afraid I am not smart enough to help, otherwise would definitely have volunteered. @p-e-w would you be the able to help with this. It will be really great to have DRY implemented for batch inferences as well.

p-e-w commented 2 months ago

I'm happy to review a pull request and answer any questions. However, I suggest waiting for a maintainer to comment on whether they actually want DRY in vLLM.

Shreyansh1311 commented 2 months ago

@p-e-w I am not sure if @AlpinDale was talking about vLLM or aphrodite-engine, but I guess it will be equally good to have it in aphrodite, if that's what AlpinDale meant.