Open Shreyansh1311 opened 1 month ago
I've been attempting to implement this, but dry_sequence_breakers has been an absolute chore, and made me somewhat give up. If someone wants to collaborate on this, please let me know. I've done most of the work, just need one final push with some issues related to that.
Oh that's really wonderful @AlpinDale. I am afraid I am not smart enough to help, otherwise would definitely have volunteered. @p-e-w would you be the able to help with this. It will be really great to have DRY implemented for batch inferences as well.
I'm happy to review a pull request and answer any questions. However, I suggest waiting for a maintainer to comment on whether they actually want DRY in vLLM.
@p-e-w I am not sure if @AlpinDale was talking about vLLM or aphrodite-engine, but I guess it will be equally good to have it in aphrodite, if that's what AlpinDale meant.
+1 absolutely need this one
need this sampler +1
+1 these samplers are important for creative writing!
+1 This would be great to have!
god we need this
I would like to see XTC, DRY, and Smoothing Factor, they have me stuck on KoboldCPP currently.
🚀 The feature, motivation and pitch
DRY is a sampler that completely mitigates repetitions. This is especially important for small models which tend to slop in large contexts. Here's an explanation of DRY from the author himself https://github.com/oobabooga/text-generation-webui/pull/5677 Along with oobabooga, koboldcpp also has an implementation of DRY which according to author IIRC is better than oobabooga's DRY has been a completely game changer for me and from what I have seen several others. It completely removes need for other samplers like top_p, top_k, repetition_penalty. It is recommended to be used with min_p and produces great coherent results.
Alternatives
No response
Additional context
No response
Before submitting a new issue...