vllm-project / vllm

A high-throughput and memory-efficient inference and serving engine for LLMs
https://docs.vllm.ai
Apache License 2.0
29.96k stars 4.52k forks source link

[Feature]: DRY Sampling #8581

Open Shreyansh1311 opened 1 month ago

Shreyansh1311 commented 1 month ago

🚀 The feature, motivation and pitch

DRY is a sampler that completely mitigates repetitions. This is especially important for small models which tend to slop in large contexts. Here's an explanation of DRY from the author himself https://github.com/oobabooga/text-generation-webui/pull/5677 Along with oobabooga, koboldcpp also has an implementation of DRY which according to author IIRC is better than oobabooga's DRY has been a completely game changer for me and from what I have seen several others. It completely removes need for other samplers like top_p, top_k, repetition_penalty. It is recommended to be used with min_p and produces great coherent results.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

AlpinDale commented 1 month ago

I've been attempting to implement this, but dry_sequence_breakers has been an absolute chore, and made me somewhat give up. If someone wants to collaborate on this, please let me know. I've done most of the work, just need one final push with some issues related to that.

Shreyansh1311 commented 1 month ago

Oh that's really wonderful @AlpinDale. I am afraid I am not smart enough to help, otherwise would definitely have volunteered. @p-e-w would you be the able to help with this. It will be really great to have DRY implemented for batch inferences as well.

p-e-w commented 1 month ago

I'm happy to review a pull request and answer any questions. However, I suggest waiting for a maintainer to comment on whether they actually want DRY in vLLM.

Shreyansh1311 commented 1 month ago

@p-e-w I am not sure if @AlpinDale was talking about vLLM or aphrodite-engine, but I guess it will be equally good to have it in aphrodite, if that's what AlpinDale meant.

azulika commented 1 month ago

+1 absolutely need this one

ZeroYuJie commented 1 month ago

need this sampler +1

KevinAHM commented 3 weeks ago

+1 these samplers are important for creative writing!

Joe986532 commented 3 weeks ago

+1 This would be great to have!

Sergal3 commented 3 weeks ago

god we need this

canineassbandit commented 1 week ago

I would like to see XTC, DRY, and Smoothing Factor, they have me stuck on KoboldCPP currently.