Closed LeonKalt closed 2 weeks ago
timoklimmer
commented
2 weeks ago The pool timeout is set/increased to 15 seconds by default. I will add a feature to customize timeouts, but my gut feeling is that you have a capacity constraint somewhere, which should be fixed rather than simply increasing timeouts. Do you have an idea what could cause httpx to run into the timeout? Are you running into network bandwidth issues?
timoklimmer
commented
2 weeks ago @LeonKalt I have added an option to configure the connection limits and timeouts when connecting to Azure OpenAI and made it part of the latest release just released. Closing the issue for now as there not more I can do here. It might be worth for you to investigate why you are running into the timeouts.
LeonKalt
commented
2 weeks ago @timoklimmer I will update to v.0.10.4 and will get back to you if we encounter more problems. Thanks for the help!
Currently we face httpx.PoolTimeouts on version v0.10.3 ` Traceback (most recent call last):
| 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi | 2024-06-19T07:01:54.493Z | result = await app( # type: ignore[func-returns-value] | 2024-06-19T07:01:54.493Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call | 2024-06-19T07:01:54.493Z | await super().call(scope, receive, send) | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in call | 2024-06-19T07:01:54.493Z | await self.middleware_stack(scope, receive, send) | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call | 2024-06-19T07:01:54.493Z | raise exc | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call | 2024-06-19T07:01:54.493Z | await self.app(scope, receive, _send) | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call | 2024-06-19T07:01:54.493Z | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app | 2024-06-19T07:01:54.493Z | raise exc | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app | 2024-06-19T07:01:54.493Z | await app(scope, receive, sender) | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in call | 2024-06-19T07:01:54.493Z | await self.middleware_stack(scope, receive, send) | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app | 2024-06-19T07:01:54.493Z | await route.handle(scope, receive, send) | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle | 2024-06-19T07:01:54.493Z | await self.app(scope, receive, send) | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app | 2024-06-19T07:01:54.493Z | await wrap_app_handling_exceptions(app, request)(scope, receive, send) | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app | 2024-06-19T07:01:54.493Z | raise exc | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app | 2024-06-19T07:01:54.493Z | await app(scope, receive, sender) | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app | 2024-06-19T07:01:54.493Z | response = await func(request) | 2024-06-19T07:01:54.493Z | ^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app | 2024-06-19T07:01:54.493Z | raw_response = await run_endpoint_function( | 2024-06-19T07:01:54.493Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:01:54.493Z | File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function | 2024-06-19T07:01:54.493Z | return await dependant.call(values) | 2024-06-19T07:01:54.493Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:01:54.493Z | File "/app/powerproxy.py", line 361, in handle_request | 2024-06-19T07:01:54.494Z | aoai_response = await aoai_target["endpoint_client"].send( | 2024-06-19T07:01:54.494Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:01:54.494Z | File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1661, in send | 2024-06-19T07:01:54.494Z | response = await self._send_handling_auth( | 2024-06-19T07:01:54.494Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:01:54.494Z | File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1689, in _send_handling_auth | 2024-06-19T07:01:54.494Z | response = await self._send_handling_redirects( | 2024-06-19T07:01:54.494Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:01:54.494Z | File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1726, in _send_handling_redirects | 2024-06-19T07:01:54.494Z | response = await self._send_single_request(request) | 2024-06-19T07:01:54.494Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:01:54.494Z | File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1763, in _send_single_request | 2024-06-19T07:01:54.494Z | response = await transport.handle_async_request(request) | 2024-06-19T07:01:54.494Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:01:54.494Z | File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 372, in handle_async_request | 2024-06-19T07:01:54.494Z | with map_httpcore_exceptions(): | 2024-06-19T07:01:54.494Z | File "/usr/local/lib/python3.11/contextlib.py", line 158, in exit | 2024-06-19T07:01:54.494Z | self.gen.throw(typ, value, traceback) | 2024-06-19T07:01:54.494Z | File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions | 2024-06-19T07:01:54.494Z | raise mapped_exc(message) from exc | 2024-06-19T07:01:54.494Z | httpx.PoolTimeout | 2024-06-19T07:02:08.057Z | ERROR: Exception in ASGI application | 2024-06-19T07:02:08.057Z | Traceback (most recent call last): | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 69, in map_httpcore_exceptions | 2024-06-19T07:02:08.057Z | yield | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 373, in handle_async_request | 2024-06-19T07:02:08.057Z | resp = await self._pool.handle_async_request(req) | 2024-06-19T07:02:08.057Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 216, in handle_async_request | 2024-06-19T07:02:08.057Z | raise exc from None | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 192, in handle_async_request | 2024-06-19T07:02:08.057Z | connection = await pool_request.wait_for_connection(timeout=timeout) | 2024-06-19T07:02:08.057Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/httpcore/_async/connection_pool.py", line 35, in wait_for_connection | 2024-06-19T07:02:08.057Z | await self._connection_acquired.wait(timeout=timeout) | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/httpcore/_synchronization.py", line 148, in wait | 2024-06-19T07:02:08.057Z | with map_exceptions(anyio_exc_map): | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/contextlib.py", line 158, in exit | 2024-06-19T07:02:08.057Z | self.gen.throw(typ, value, traceback) | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/httpcore/_exceptions.py", line 14, in map_exceptions | 2024-06-19T07:02:08.057Z | raise to_exc(exc) from exc | 2024-06-19T07:02:08.057Z | httpcore.PoolTimeout | 2024-06-19T07:02:08.057Z | The above exception was the direct cause of the following exception: | 2024-06-19T07:02:08.057Z | Traceback (most recent call last): | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi | 2024-06-19T07:02:08.057Z | result = await app( # type: ignore[func-returns-value] | 2024-06-19T07:02:08.057Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call | 2024-06-19T07:02:08.057Z | await super().call(scope, receive, send) | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in call | 2024-06-19T07:02:08.057Z | await self.middleware_stack(scope, receive, send) | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call | 2024-06-19T07:02:08.057Z | raise exc | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call | 2024-06-19T07:02:08.057Z | await self.app(scope, receive, _send) | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call | 2024-06-19T07:02:08.057Z | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app | 2024-06-19T07:02:08.057Z | raise exc | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app | 2024-06-19T07:02:08.057Z | await app(scope, receive, sender) | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in call | 2024-06-19T07:02:08.057Z | await self.middleware_stack(scope, receive, send) | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app | 2024-06-19T07:02:08.057Z | await route.handle(scope, receive, send) | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle | 2024-06-19T07:02:08.057Z | await self.app(scope, receive, send) | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app | 2024-06-19T07:02:08.057Z | await wrap_app_handling_exceptions(app, request)(scope, receive, send) | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app | 2024-06-19T07:02:08.057Z | raise exc | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app | 2024-06-19T07:02:08.057Z | await app(scope, receive, sender) | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app | 2024-06-19T07:02:08.057Z | response = await func(request) | 2024-06-19T07:02:08.057Z | ^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app | 2024-06-19T07:02:08.057Z | raw_response = await run_endpoint_function( | 2024-06-19T07:02:08.057Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function | 2024-06-19T07:02:08.057Z | return await dependant.call(values) | 2024-06-19T07:02:08.057Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:02:08.057Z | File "/app/powerproxy.py", line 361, in handle_request | 2024-06-19T07:02:08.057Z | aoai_response = await aoai_target["endpoint_client"].send( | 2024-06-19T07:02:08.057Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1661, in send | 2024-06-19T07:02:08.057Z | response = await self._send_handling_auth( | 2024-06-19T07:02:08.057Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1689, in _send_handling_auth | 2024-06-19T07:02:08.057Z | response = await self._send_handling_redirects( | 2024-06-19T07:02:08.057Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1726, in _send_handling_redirects | 2024-06-19T07:02:08.057Z | response = await self._send_single_request(request) | 2024-06-19T07:02:08.057Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1763, in _send_single_request | 2024-06-19T07:02:08.057Z | response = await transport.handle_async_request(request) | 2024-06-19T07:02:08.057Z | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 372, in handle_async_request | 2024-06-19T07:02:08.057Z | with map_httpcore_exceptions(): | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/contextlib.py", line 158, in exit | 2024-06-19T07:02:08.057Z | self.gen.throw(typ, value, traceback) | 2024-06-19T07:02:08.057Z | File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions | 2024-06-19T07:02:08.057Z | raise mapped_exc(message) from exc | 2024-06-19T07:02:08.057Z | httpx.PoolTimeout
`