neurobagel / federation-api

REST API for query federation over multiple Neurobagel graph databases
https://federate.neurobagel.org
MIT License
0 stars 0 forks source link

[BUG] Node-specific `ConnectTimeout` not caught properly during federated query #72

Closed alyssadai closed 8 months ago

alyssadai commented 9 months ago

Is there an existing issue for this?

Expected Behavior

When one of the nodes that a federated query is sent to times out, we expect a successful f-API response to still be returned which includes results for the nodes for which the query succeeded, and simply wrap any timeouts/errors in the "errors" field in the f-API response.

Current Behavior

When the connection to a node times out with a ConnectTimeout, the f-API errors out with an internal server error 😢

Error message

Error from f-API logs:

INFO:     172.19.10.1:48332 - "GET /query/?is_control=true&image_modal=https%3A%2F%2Fapi-bic.neurobagel.org%2F HTTP/1.0" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 407, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "/usr/local/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/fastapi/applications.py", line 276, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/cors.py", line 83, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.10/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "/usr/local/lib/python3.10/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "/usr/src/./app/api/routers/query.py", line 14, in get_query
    response = await crud.get(
  File "/usr/src/./app/api/crud.py", line 94, in get
    for result in response:
TypeError: 'ConnectTimeout' object is not iterable

Environment

How to reproduce

No response

Anything else?

Most likely, this is because the category of request errors we're currently trying to catch (which we hoped would capture timeouts) is too narrow - it includes ConnectError, but not ConnectTimeout: see https://www.python-httpx.org/exceptions/#the-exception-hierarchy

https://github.com/neurobagel/federation-api/blob/3e76fd0e5700519c03d5aae8c6b7df6ff5d0c982/app/api/utility.py#L241-L245

We should broaden this to RequestError and see if that fixes things.

surchs commented 7 months ago

:rocket: Issue was released in v0.1.0 :rocket: