Does vllm support do_sample?

[leiwen83]

Hi,

For hugginface, it support various sample strategy: https://huggingface.co/docs/transformers/main/main_classes/text_generation

greedy decoding by calling greedy_search() if num_beams=1 and do_sample=False contrastive search by calling contrastive_search() if penalty_alpha>0. and top_k>1 multinomial sampling by calling sample() if num_beams=1 and do_sample=True beam-search decoding by calling beam_search() if num_beams>1 and do_sample=False beam-search multinomial sampling by calling beam_sample() if num_beams>1 and do_sample=True

Since vllm already support beam search with best_of parameter, I wonder how do we support the do_sample together with best_of for the beam sample strategy?

Thx

[lucasjinreal]

Same

[BaiMoHan]

I need this support🥺🥺🥺🥺🥺🥺

[abdulvirta]

same

[samarthsarin]

Any update on this feature? @WoosukKwon

[latinostats]

I am also interested on this feature.

[Ksuriuri]

Refer to vllm->model_executor->layers->sampler.py->_sample: While sampling_type == SamplingType.RANDOM, the implementation is the same as sample() in huggingface transformers.

So you just need to set the sampling type to "SamplingType.RANDOM", and refer to vllm->sampling_params.py->sampling_type: You should set use_beam_search = false and temperature > 1e-5 while you initializing SamplingParams.

[lauhaide]

Hello, what is the difference between SamplingType.RANDOM and SamplingType.RANDOM_SEED? As far as I followed the code it seems the same. I was not able to find where the 'seed' argument in SamplingParams is used to set any seed.

Any comments on this?

Thank you in advance!