getting error code 504.0 Gateway Timeout

[virajmehta6899]

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Clone the repository, set up the Azure DevOps environment, and then repeatedly test by making 3 to 4 requests to question. endpoint :- /chat

Any log messages given by the failure

[Error] Middleware: Failed to forward request to http://169.254.130.17:8000. Encountered a System.Threading.Tasks.TaskCanceledException exception after 300022.558ms with message: The request was canceled due to the configured HttpClient.Timeout of 300 seconds elapsing.. Check application logs to verify the application is properly handling HTTP traffic.

Traceback (most recent call last): 2023-08-09T07:55:51.123227865Z File "/tmp/8db98a655c3497d/app.py", line 152, in chat 2023-08-09T07:55:51.123238066Z r = impl.run(request.json["history"], request.json.get("overrides") or {}) 2023-08-09T07:55:51.123243466Z File "/tmp/8db98a655c3497d/approaches/chatreadretrieveread.py", line 83, in run 2023-08-09T07:55:51.123248466Z chat_completion = openai.ChatCompletion.create( 2023-08-09T07:55:51.123253467Z File "/tmp/8db98a655c3497d/antenv/lib/python3.10/site-packages/openai/api_resources/chat_completion.py", line 25, in create 2023-08-09T07:55:51.123258667Z return super().create(*args, **kwargs) 2023-08-09T07:55:51.123263567Z File "/tmp/8db98a655c3497d/antenv/lib/python3.10/site-packages/openai/api_resources/abstract/engine_apiresource.py", line 153, in create 2023-08-09T07:55:51.123268768Z response, , api_key = requestor.request( 2023-08-09T07:55:51.123273968Z File "/tmp/8db98a655c3497d/antenv/lib/python3.10/site-packages/openai/api_requestor.py", line 288, in request 2023-08-09T07:55:51.123279168Z result = self.request_raw( 2023-08-09T07:55:51.123284169Z File "/tmp/8db98a655c3497d/antenv/lib/python3.10/site-packages/openai/api_requestor.py", line 607, in request_raw 2023-08-09T07:55:51.123289169Z raise error.Timeout("Request timed out: {}".format(e)) from e 2023-08-09T07:55:51.123294269Z openai.error.Timeout: Request timed out: HTTPSConnectionPool(host='cog-uh6bfxy6yc7z4.openai.azure.com', port=443): Read timed out. (read timeout=600)

Expected/desired behavior

It is expected to function correctly for every request we send. However, please note that an error may arise after sending 4 to 5 requests.

OS and Version?

Windows 10.

azd version?

1.1.0.

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

[pamelafox]

Please see the discussion on https://github.com/Azure-Samples/azure-search-openai-demo/issues/420 as I suspect your timeout is related to our discussions there. We are experimenting with various approaches to dealing with OpenAI timeouts, like specifying request_timeout in ChatCompletion.create() or using the tenacity package for retries. You can also try a different region, as some OpenAI regions (like southcentralus) are experiencing more load related issues than others.

[GitTxb2]

Thanks @pamelafox. Do you think all the dependencies on the repo would work if I migrated the entire resource group to Australia East? https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#model-summary-table-and-region-availability

[pamelafox]

Actually, no, embeddings arent available there. See my recent PR for what regions would work for both: https://github.com/Azure-Samples/azure-search-openai-demo/pull/520 You could also make separate parameters for embedding, if you really want to use Australia East for gpt35.

[github-actions[bot]]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this issue will be closed.