Java: .Net: New Feature: rate limit metadata

microsoft / semantic-kernel

Integrate cutting-edge LLM technology quickly and easily into your apps

https://aka.ms/semantic-kernel

MIT License

21.4k stars 3.15k forks source link

Java: .Net: New Feature: rate limit metadata #8421

Open mku-wedoai opened 3 weeks ago

mku-wedoai commented 3 weeks ago

Raw Azure OpenAi chat completion response contains additional headers, which contain information about rate limits: x-rate-limit-remaining-requests and x-ratelimit-remaining-tokens. Would it be possible to add these values to the Metadata dictionary in both OpenAIStreamingChatMessageContent and OpenAIChatMessageContent class? I find rate limit information useful, but in current .Net implementation there is no way to access it.

markwallace-microsoft commented 3 weeks ago

@d3r3kk @johnoliver @moonbox3 We should implement this in Java and Python also