When one of the nodes that a federated query is sent to times out, we expect a successful f-API response to still be returned which includes results for the nodes for which the query succeeded, and simply wrap any timeouts/errors in the "errors" field in the f-API response.
Current Behavior
When the connection to a node times out with a ConnectTimeout, the f-API errors out with an internal server error 😢
Error message
Error from f-API logs:
INFO: 172.19.10.1:48332 - "GET /query/?is_control=true&image_modal=https%3A%2F%2Fapi-bic.neurobagel.org%2F HTTP/1.0" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
return await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 276, in __call__
await super().__call__(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
raise exc
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 83, in __call__
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
raise exc
File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
await self.app(scope, receive, sender)
File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
raise e
File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
response = await func(request)
File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 237, in app
raw_response = await run_endpoint_function(
File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
return await dependant.call(**values)
File "/usr/src/./app/api/routers/query.py", line 14, in get_query
response = await crud.get(
File "/usr/src/./app/api/crud.py", line 94, in get
for result in response:
TypeError: 'ConnectTimeout' object is not iterable
Environment
OS:
Python/Node version:
How to reproduce
No response
Anything else?
Most likely, this is because the category of request errors we're currently trying to catch (which we hoped would capture timeouts) is too narrow - it includes ConnectError, but not ConnectTimeout: see https://www.python-httpx.org/exceptions/#the-exception-hierarchy
Is there an existing issue for this?
Expected Behavior
When one of the nodes that a federated query is sent to times out, we expect a successful f-API response to still be returned which includes results for the nodes for which the query succeeded, and simply wrap any timeouts/errors in the
"errors"
field in the f-API response.Current Behavior
When the connection to a node times out with a
ConnectTimeout
, the f-API errors out with aninternal server error
😢Error message
Error from f-API logs:
Environment
How to reproduce
No response
Anything else?
Most likely, this is because the category of request errors we're currently trying to catch (which we hoped would capture timeouts) is too narrow - it includes
ConnectError
, but notConnectTimeout
: see https://www.python-httpx.org/exceptions/#the-exception-hierarchyhttps://github.com/neurobagel/federation-api/blob/3e76fd0e5700519c03d5aae8c6b7df6ff5d0c982/app/api/utility.py#L241-L245
We should broaden this to
RequestError
and see if that fixes things.