Productionizing the application for 1000 users

microsoft / sample-app-aoai-chatGPT

Sample code for a simple web chat experience through Azure OpenAI, including Azure OpenAI On Your Data.

MIT License

1.64k stars 2.58k forks source link

Productionizing the application for 1000 users #526

Open npsquared opened 9 months ago

npsquared commented 9 months ago

Any tips or recommendations on productionizing this app for a 1000 users? I found this guide in the other repo: https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/productionizing.md

Anyone who was able to successfully put this in production, please comment.

Thank you.

log2 commented 9 months ago

Hello @npsquared, I have the same aim for the app in my case, so I'm looking at not-ready-for-production pitfalls in order to remove them.

I think that some parts of the application should be looked upon carefully; for example, passing the message_uuid of the assistant message being build using a global variable is fragile (see here for the variable definition, here for the assignment performed in a service, but used in the context of other services such as content streaming).

sarah-widder commented 9 months ago

Hi @npsquared @log2 #460 was merged yesterday and aims to prepare the app for production by refactoring to async. This would allow for many more simultaneous users. We are also looking into refactoring the global message_uuid. If you have additional feedback or suggestions please do let us know :)

sarah-widder commented 8 months ago

Update - #623 removed the global message_uuid

npsquared commented 8 months ago

Thanks for the amazing work, @sarah-widder! I have incorporated the async refactor and the global message_uuid update to my fork. So far so good. I am looking to put this in production by the end of next week. I will start a discussion post at some point to summarize our experience putting this app in production.

Any other thoughts, @log2 ?

sgkalyans commented 3 months ago

Hi, the cosmos DB returning 400 bad requests when application been accessed by many users at the same time but no issue when accessed by 1 or 2 users at the same time. The error occurred in "create_message" method.