Open gabrielhuang opened 1 month ago
In case it helps others, using max_tokens=1
solved the issue
Actually let me reopen this since it appears that we're supposed to specifically allow max_tokens=0
when echo=True
.
https://github.com/vllm-project/vllm/blob/main/vllm/entrypoints/openai/protocol.py#L399
Your current environment
🐛 Describe the bug
I'm trying to get the logprobability of the last token (Yes or No) of the prompt.
In my client, I set:
However, as soon as i query the vLLM server, it crashes with the following message inthe logs