[Bug] Models generate whitespace-only output when temperature is in range [1e-4, 1e-5], regardless of model type

[saumya-saran]

Setting the temperature in a particular range causes vllm to generate whitespace-only outputs. Values above/below this range work correctly. I have seen this with facebook/opt-125m, fine-tuned mistral-7B models, codellama-13B, and several other models. It seems like this is an issue with vllm rather than the particular model:

To reproduce: python -m vllm.entrypoints.openai.api_server --model facebook/opt-125m

Send request:

curl http://localhost:8000/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "facebook/opt-125m",
"prompt": "San Francisco is a",
"max_tokens": 7,
"temperature": <temperature>
}'

With temperature: 1e-3: Generates " great place to live. I" 1e-4: Generates "\\\\\\\" 1e-5: Generates "\\\\\\\" 1e-6: Generates " great place to live. I"

[hmellor]

The reason it starts working again at 1e-6 is because it is less than _SAMPLING_EPS, so it reverts to greedy sampling:

https://github.com/vllm-project/vllm/blob/b98cc28f91aadbb8b831611f3676da92f892211d/vllm/sampling_params.py#L15

https://github.com/vllm-project/vllm/blob/b98cc28f91aadbb8b831611f3676da92f892211d/vllm/sampling_params.py#L254-L259

[hmellor]

Looks like this was fixed in #2937 with:

https://github.com/vllm-project/vllm/blob/b98cc28f91aadbb8b831611f3676da92f892211d/vllm/sampling_params.py#L16

https://github.com/vllm-project/vllm/blob/b98cc28f91aadbb8b831611f3676da92f892211d/vllm/sampling_params.py#L220-L225