search

openai / openai-dotnet

The official .NET library for the OpenAI API
https://www.nuget.org/packages/OpenAI
MIT License
707 stars 60 forks source link

Exceeding rate limits results in a retry which ends with 401 Unauthorized #95

Closed molinch closed 2 days ago

molinch commented 2 days ago

We have a rather big prompt, and a small rate limit of 1000 tokens/minute. Due to that combination we can invoke the OpenAPI endpoint only once every minute.

If we exceed that we get back such an exception:

"message": "Service request failed.\nStatus: 401 (Unauthorized)\n",
"stackTrace": "   at Azure.AI.OpenAI.ClientPipelineExtensions.ProcessMessageAsync(ClientPipeline pipeline, PipelineMessage message, RequestOptions options)\n   at Azure.AI.OpenAI.Chat.AzureChatClient.CompleteChatAsync(BinaryContent content, RequestOptions options)\n   at OpenAI.Chat.ChatClient.<>c__DisplayClass8_0.<<CompleteChatStreamingAsync>g__getResultAsync|0>d.MoveNext()\n--- End of stack trace from previous location ---\n   at OpenAI.Chat.AsyncStreamingChatCompletionUpdateCollection.AsyncStreamingChatUpdateEnumerator.CreateEventEnumeratorAsync()\n   at OpenAI.Chat.AsyncStreamingChatCompletionUpdateCollection.AsyncStreamingChatUpdateEnumerator.System.Collections.Generic.IAsyncEnumerator<OpenAI.Chat.StreamingChatCompletionUpdate>.MoveNextAsync()\n   at Sofia.Common.DigitalAssistantModule.Clients.OpenAi.OpenAiClient.GetChatCompletionsStream(String text, Language language, OpenApiUseCaseOption options, CancellationToken cancellationToken, String callerMemberName)+MoveNext() in /src/Sofia.Common.DigitalAssistantModule/Clients/OpenAi/OpenAiClient.cs:line 125\n   at Sofia.Common.DigitalAssistantModule.Clients.OpenAi.OpenAiClient.GetChatCompletionsStream(String text, Language language, OpenApiUseCaseOption options, CancellationToken cancellationToken, String callerMemberName)+MoveNext() in /src/Sofia.Common.DigitalAssistantModule/Clients/OpenAi/OpenAiClient.cs:line 125\n   at Sofia.Common.DigitalAssistantModule.Clients.OpenAi.OpenAiClient.GetChatCompletionsStream(String text, Language language, OpenApiUseCaseOption options, CancellationToken cancellationToken, String callerMemberName)

image

We investigated deeper and the truth is that at first there is a 429 (due to rate limits), then I assume the client retries, which results in a 401, and then this exception.

It leads to very wrong investigations, as you think you have a 401, when it's actually a rate limiting issue. Now the big question would be why upon retrying it becomes unauthorized.

joseharriaga commented 2 days ago

Thank you for reaching out, @molinch ! Looking at the stack trace, I see that you are using the Azure.AI.OpenAI library. This repo corresponds to the OpenAI library. Could you file this issue in the Azure repo instead? Here's a link: 🔗 https://github.com/Azure/azure-sdk-for-net

When you do, including the following information would be helpful:

  1. You are using the Azure.AI.OpenAI library, but what service are you communicating with: OpenAI or Azure OpenAI?
  2. If you are using the Azure OpenAI service, what auth mechanism are you using?
molinch commented 2 days ago

Thanks a lot @joseharriaga, this feedback really helped me, I moved it there, https://github.com/Azure/azure-sdk-for-net/issues/44804 I will close this issue then