search

sgl-project / sglang

SGLang is a fast serving framework for large language models and vision language models.
https://sglang.readthedocs.io/en/latest/
Apache License 2.0
5.12k stars 360 forks source link

[Feature] Does sglang now support beam search #903

Open StevenZHB opened 1 month ago

StevenZHB commented 1 month ago

Motivation

Does sglang now support beam search? I tried using the 'best_of' parameter, but the output seems to be the same as if it wasn't used.

Related resources

No response

zhyncs commented 1 month ago

Currently not supported. Whether to support it in the future needs to be discussed. At present, other features and optimizations may have a higher priority. Thank you for your interest in SGLang. Please stay tuned, thanks.

min-xu-et commented 1 month ago

I agree it is a low priority. There are a lot of way to generate diverse output from LLMs and then select a good response. Unclear that beam search is a good approach anyway.