Open epark001 opened 1 month ago
Normal LLM returns the CompletionUsage data in the last chunk:
Based on the request ID, it seems that you are using the Completions API rather than Chat Completions API for normal LLM. To better narrow down the issue, can you try using Chat Completions API for normal LLM as well?
@epark001 I think you need to include stream_options
including include_usage=True
when you send the request, since that is not enabled by default in vLLM.
Here is an example via the openai python client:
from openai import OpenAI
# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"
client = OpenAI(
# defaults to os.environ.get("OPENAI_API_KEY")
api_key=openai_api_key,
base_url=openai_api_base,
)
models = client.models.list()
model = models.data[0].id
chat_completion = client.chat.completions.create(
messages=[{
"role": "system",
"content": "You are a helpful assistant."
}, {
"role": "user",
"content": "Who won the world series in 2020?"
}, {
"role":
"assistant",
"content":
"The Los Angeles Dodgers won the World Series in 2020."
}, {
"role": "user",
"content": "Where was it played?"
}],
model=model,
max_tokens=10,
stream=True,
stream_options={
'include_usage': True,
}
)
for c in chat_completion:
print(c)
produces:
ChatCompletionChunk(id='chat-0a9c17690faf42f899bad7ecf0d61d30', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, role='assistant', tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721845881, model='facebook/opt-125m', object='chat.completion.chunk', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chat-0a9c17690faf42f899bad7ecf0d61d30', choices=[Choice(delta=ChoiceDelta(content='The', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721845881, model='facebook/opt-125m', object='chat.completion.chunk', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chat-0a9c17690faf42f899bad7ecf0d61d30', choices=[Choice(delta=ChoiceDelta(content=' only', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721845881, model='facebook/opt-125m', object='chat.completion.chunk', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chat-0a9c17690faf42f899bad7ecf0d61d30', choices=[Choice(delta=ChoiceDelta(content=' thing', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721845881, model='facebook/opt-125m', object='chat.completion.chunk', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chat-0a9c17690faf42f899bad7ecf0d61d30', choices=[Choice(delta=ChoiceDelta(content=' I', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721845881, model='facebook/opt-125m', object='chat.completion.chunk', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chat-0a9c17690faf42f899bad7ecf0d61d30', choices=[Choice(delta=ChoiceDelta(content=' don', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721845881, model='facebook/opt-125m', object='chat.completion.chunk', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chat-0a9c17690faf42f899bad7ecf0d61d30', choices=[Choice(delta=ChoiceDelta(content="'t", function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721845881, model='facebook/opt-125m', object='chat.completion.chunk', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chat-0a9c17690faf42f899bad7ecf0d61d30', choices=[Choice(delta=ChoiceDelta(content=' understand', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721845881, model='facebook/opt-125m', object='chat.completion.chunk', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chat-0a9c17690faf42f899bad7ecf0d61d30', choices=[Choice(delta=ChoiceDelta(content=' is', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721845881, model='facebook/opt-125m', object='chat.completion.chunk', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chat-0a9c17690faf42f899bad7ecf0d61d30', choices=[Choice(delta=ChoiceDelta(content=' why', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1721845881, model='facebook/opt-125m', object='chat.completion.chunk', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chat-0a9c17690faf42f899bad7ecf0d61d30', choices=[Choice(delta=ChoiceDelta(content=' it', function_call=None, role=None, tool_calls=None), finish_reason='length', index=0, logprobs=None, stop_reason=None)], created=1721845881, model='facebook/opt-125m', object='chat.completion.chunk', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='chat-0a9c17690faf42f899bad7ecf0d61d30', choices=[], created=1721845881, model='facebook/opt-125m', object='chat.completion.chunk', system_fingerprint=None, usage=CompletionUsage(completion_tokens=10, prompt_tokens=34, total_tokens=44))
The usage is there as expected.
One thing to note is that this does look like a regression in default behavior from vllm v0.4.3 to v0.5.0
It seems like the default behavior was to include it in the last chunk in v0.4.3. In v0.5.0 there is an additional chunk with the usage if stream_options is set.
Within vllm/entrypoints/openai/serving_completion.py#completion_stream_generator and serving_chat.py#chat_completion_stream_generator
On the left is v0.5.0, the right has v0.4.3
Would it make sense to reinstate this default behavior? Or add an environment variable that would allow it? This difference in default behavior is impacting me and my team.
One thing to note is that this does look like a regression in default behavior from vllm v0.4.3 to v0.5.0
It seems like the default behavior was to include it in the last chunk in v0.4.3. In v0.5.0 there is an additional chunk with the usage if stream_options is set.
Within vllm/entrypoints/openai/serving_completion.py#completion_stream_generator and serving_chat.py#chat_completion_stream_generator
On the left is v0.5.0, the right has v0.4.3
Would it make sense to reinstate this default behavior? Or add an environment variable that would allow it? This difference in default behavior is impacting me and my team.
Does setting include_usage=True
work for you? Btw, this behaviour is specified by OpenAI, so we shouldn't deviate from it.
Setting it in the request headers or when using the OpenAI Python SDK does work.
Totally understand wanting to follow the OpenAI spec.
Our team's issue is that we sit in-between our customers using the SDK and the instances of vllm we host on prem. We were using the usage token count for rate limiting purposes, and would rather not inject include_usage into our customer's requests.
Your current environment
🐛 Describe the bug
When streaming chat completions for VLM (I am using phi-3-vision) no CompletionUsage is returned in the last chunk, but normal non-streaming chat completions outputs the CompletionUsage token counts. Normal:
Streaming
Normal LLM returns the CompletionUsage data in the last chunk: