search

hyperonym / basaran

Basaran is an open-source alternative to the OpenAI text completion API. It provides a compatible streaming API for your Hugging Face Transformers-based text generation models.
MIT License
1.29k stars 80 forks source link

Beam search #97

Closed lcw99 closed 1 year ago

lcw99 commented 1 year ago

I am using this package, which shows much better performance than MS DeepSpeed-MII, which has a similar function. Do you have plans to implement beam search as well?

peakji commented 1 year ago

Currently we do not have plans to implement beam search. On one hand, it would incur greater performance overhead, and on the other hand, if you want to generate natural text, beam search may not be the best choice.

lcw99 commented 1 year ago

Thanks for your answer. I really expecting Frequency and presence penalties functionality. Currently, I have some repetation problem on my model.