search

Azure-Samples / azure-search-openai-demo

A sample app for the Retrieval-Augmented Generation pattern running in Azure, using Azure AI Search for retrieval and Azure OpenAI large language models to power ChatGPT-style and Q&A experiences.
https://azure.microsoft.com/products/search
MIT License
5.62k stars 3.76k forks source link

Add optional APIM (API Management) to infrastructure #817

Open pamelafox opened 8 months ago

pamelafox commented 8 months ago

We're discussing APIM integration in this sample, as we know many developers are manually adding APIM, so we'd like to make it easier for everyone.

However, there are many ways to use APIM, so we want to clarify what is most popular with developers.

If you're interested in APIM, please clarify:

We'd love feedback, thanks!

vrajroutu commented 8 months ago

The load balancing between instances is performing exceptionally well, even with over 3000+ users. I implemented this solution using the guidelines provided in the reference article:

Azure OpenAI Load Balancing.

jws-m-gotoh commented 8 months ago

Thanks for the great code. In my case, I want to enforce authentication using APIM. APIM is placed in front of the OpenAI endpoint.

UmarMohamedUsman commented 7 months ago

@pamelafox, all the uses cases you mentioned are really important. My priority order :)

  1. Load balancing across multiple AOAI instances (very critical)
  2. Rate limiting
  3. Enforcing authentication
  4. Monitoring token usage

APIM in front of AOAI endpoint should be good enough.

adamhrin commented 2 months ago

Hello, thanks for the repo, we appreciate the effort!

We are interested to put this into production and put APIM in front of the entire backend. Could you point us to what changes would need to be done in the code to enable APIM for authentication (via Azure AD)?

Is it possible to enable authentication through APIM, and if so how do we need to configure the authentication setup compared to your instruction with Azure AD apps? We are interested in this so that we can for example restrict IP addresses and enable certificates.

Thank you for your help!