search

microsoft / PubSec-Info-Assistant

Information Assistant, built with Azure OpenAI Service, Industry Accelerator
MIT License
266 stars 525 forks source link

Read Time Out after Batch Testing #654

Closed deepakas closed 2 months ago

deepakas commented 2 months ago

Bug Details

Describe the bug We are using the backend API to generate response for a set of test questions. When we run for high batches the application fails.

We ran the api for 100 questions with batch size of 10 and time delay of 20 seconds. It worked 2 times. Third time application crashed. API requests to batch endpoint is giving an empty response. Web app returns this error - Error: HTTPSConnectionPool(host='infoasst-enrichmentweb-clc9v.azurewebsites.net', port=443): Read timed out. (read timeout=60) Steps To Reproduce

  1. Run Batch Testing on backend with 100 requests with a batch of 10 requests at a time.
  2. After multiple attempts the application crashes.
  3. Backend responds with empty response

What is the expected behavior? Return json response

Screenshots image ErrorsFromAppServiceLogs.txt

If applicable, add screenshots to help explain your problem.

Information Assistant details

Please provide the following details. You can simply include a screenshot of your Info panel as well.

GitHub branch: main

Version or Latest commit: commit f4b3f3ab3bc82b6ee78b024f5fad5aa9a4ee8f2d (HEAD -> main, origin/main, origin/HEAD) Merge: ce46332 a575426 Author: dayland 48474707+dayland@users.noreply.github.com What region is your Azure Open AI Service in? US east

What ChatGPT model are you using? gpt-4

model name: gpt-4

model version: 0125-preview

What embeddings model are you using? [text-embedding-ada-002]

Additional context Errors from App Service Logs :

If the bug is confirmed, would you be willing to submit a PR?

ArpitaisAn0maly commented 2 months ago

Hi deepakas. How are you running batch of questions against IA? Are they not dependent on each other?( Conversational style) If you have set of questions and pairs for 100 questions would you please run this through promptflow as flow automates this for testing.

deepakas commented 2 months ago

Thanks @ArpitaisAn0maly. I am making an api call to the backend chat api to get bot response and submitting batches of 10 and having delay of 20 seconds with retries with doubling the delay time. Resolved the issue by scaling the apps and restarting. Thanks George for your help and reference documentation. https://github.com/microsoft/PubSec-Info-Assistant/blob/main/docs/deployment/considerations_production.md

deepakas commented 2 months ago

Resolved the issue with scaling and restarting the apps - https://github.com/microsoft/PubSec-Info-Assistant/blob/main/docs/deployment/considerations_production.md