npsquared
commented
6 months ago @p-n-rao The following PR provides some insight into setting up a back-off mechanism for rate-limit errors, might be useful to you: https://github.com/Azure-Samples/azure-search-openai-demo/pull/500
I am working on setting something similar to this for our app.
Is your feature request related to this sample app, or to an Azure service, such as Azure OpenAI or Azure AI Search? Describe the scope of your request. This forum is focused on features relating to this sample app and its functionality, not the functionality of underlying Azure services. Yes. Is it possible to expand this sample APP to add APIM between APP and openAI? With this integration, we can use all the features of APIM (retry, load balance etc ...)
Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] With this App, we can get responses only <1000 request in 10 Minutes without any errors. With APIM, we can add retry logic to reduce the 429 errors
Describe the solution you'd like A clear and concise description of what you want to happen. Adding APIM between APP and openAI
Is this feature specific to your use case or your organization, or would it apply broadly across other uses of this app? Describe how you and others might leverage this feature.
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context Add any other context or screenshots about the feature request here.