Python SDK, Proxy Server (LLM Gateway) to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq]
13.57k
stars
1.59k
forks
source link
[Feature]: Use stream_options on Azure OpenAI #5751
Closed
Manouchehri closed 1 month ago
The Feature
It's undocumented, but
2024-08-01-preview
supports"stream_options": {"include_usage": true}}
.Example:
Motivation, pitch
Allows getting perfectly accurate token results in streaming responses.
Twitter / LinkedIn details
https://www.linkedin.com/in/davidmanouchehri/