search

factly / hunting

Hunting is data profiling exposed as REST API. It is built on top of Pandas Profiling & Polars.
4 stars 1 forks source link

Handle File not found error for s3 #21

Open 100mi opened 1 year ago

100mi commented 1 year ago

Description

When file is not present , then hunting server is popping out Internal Error with the following Traceback: 

hunting-hunting-server-1  | INFO:     172.31.0.1:56404 - "GET /api/v1/profile/description/?source=s3%3A%2F%2Fdgca%2Fprocessed%2Fquaterly-int-traffic-stats%2Fintl-traffic-stats-citypair%2Foutput.csv&minimal=true&samples_to_show=10 HTTP/1.1" 500 Internal Server Error
hunting-hunting-server-1  | ERROR:    Exception in ASGI application
hunting-hunting-server-1  | Traceback (most recent call last):
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/anyio/streams/memory.py", line 94, in receive
hunting-hunting-server-1  |     return self.receive_nowait()
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/anyio/streams/memory.py", line 89, in receive_nowait
hunting-hunting-server-1  |     raise WouldBlock
hunting-hunting-server-1  | anyio.WouldBlock
hunting-hunting-server-1  | 
hunting-hunting-server-1  | During handling of the above exception, another exception occurred:
hunting-hunting-server-1  | 
hunting-hunting-server-1  | Traceback (most recent call last):
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 43, in call_next
hunting-hunting-server-1  |     message = await recv_stream.receive()
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/anyio/streams/memory.py", line 114, in receive
hunting-hunting-server-1  |     raise EndOfStream
hunting-hunting-server-1  | anyio.EndOfStream
hunting-hunting-server-1  | 
hunting-hunting-server-1  | During handling of the above exception, another exception occurred:
hunting-hunting-server-1  | 
hunting-hunting-server-1  | Traceback (most recent call last):
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 419, in run_asgi
hunting-hunting-server-1  |     result = await app(  # type: ignore[func-returns-value]
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
hunting-hunting-server-1  |     return await self.app(scope, receive, send)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/applications.py", line 270, in __call__
hunting-hunting-server-1  |     await super().__call__(scope, receive, send)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/applications.py", line 124, in __call__
hunting-hunting-server-1  |     await self.middleware_stack(scope, receive, send)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 184, in __call__
hunting-hunting-server-1  |     raise exc
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py", line 162, in __call__
hunting-hunting-server-1  |     await self.app(scope, receive, _send)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 72, in __call__
hunting-hunting-server-1  |     response = await self.dispatch_func(request, call_next)
hunting-hunting-server-1  |   File "/app/./app/main.py", line 26, in add_process_time_header
hunting-hunting-server-1  |     response = await call_next(request)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 46, in call_next
hunting-hunting-server-1  |     raise app_exc
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/base.py", line 36, in coro
hunting-hunting-server-1  |     await self.app(scope, request.receive, send_stream.send)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/cors.py", line 84, in __call__
hunting-hunting-server-1  |     await self.app(scope, receive, send)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 75, in __call__
hunting-hunting-server-1  |     raise exc
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py", line 64, in __call__
hunting-hunting-server-1  |     await self.app(scope, receive, sender)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
hunting-hunting-server-1  |     raise e
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
hunting-hunting-server-1  |     await self.app(scope, receive, send)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 680, in __call__
hunting-hunting-server-1  |     await route.handle(scope, receive, send)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 275, in handle
hunting-hunting-server-1  |     await self.app(scope, receive, send)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/starlette/routing.py", line 65, in app
hunting-hunting-server-1  |     response = await func(request)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 231, in app
hunting-hunting-server-1  |     raw_response = await run_endpoint_function(
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
hunting-hunting-server-1  |     return await dependant.call(**values)
hunting-hunting-server-1  |   File "/app/./app/api/api_v1/routers/profile.py", line 211, in profile_description
hunting-hunting-server-1  |     return await get_profile(
hunting-hunting-server-1  |   File "/app/./app/utils/profile_db.py", line 113, in get_profile
hunting-hunting-server-1  |     description = await save_profile(url, minimal, samples_to_show)
hunting-hunting-server-1  |   File "/app/./app/utils/profile_db.py", line 28, in save_profile
hunting-hunting-server-1  |     dataframe = await get_dataframe_async(url)
hunting-hunting-server-1  |   File "/app/./app/utils/dataframes.py", line 136, in get_dataframe_async
hunting-hunting-server-1  |     with fs.open(f"{url.netloc}{url.path}", "rb") as f:
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/fsspec/spec.py", line 1106, in open
hunting-hunting-server-1  |     f = self._open(
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/s3fs/core.py", line 640, in _open
hunting-hunting-server-1  |     return S3File(
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/s3fs/core.py", line 1989, in __init__
hunting-hunting-server-1  |     super().__init__(
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/fsspec/spec.py", line 1462, in __init__
hunting-hunting-server-1  |     self.size = self.details["size"]
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/fsspec/spec.py", line 1475, in details
hunting-hunting-server-1  |     self._details = self.fs.info(self.path)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/fsspec/asyn.py", line 113, in wrapper
hunting-hunting-server-1  |     return sync(self.loop, func, *args, **kwargs)
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/fsspec/asyn.py", line 98, in sync
hunting-hunting-server-1  |     raise return_result
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/fsspec/asyn.py", line 53, in _runner
hunting-hunting-server-1  |     result[0] = await coro
hunting-hunting-server-1  |   File "/usr/local/lib/python3.9/site-packages/s3fs/core.py", line 1257, in _info
hunting-hunting-server-1  |     raise FileNotFoundError(path)
hunting-hunting-server-1  | FileNotFoundError: dgca/processed/quaterly-int-traffic-stats/intl-traffic-stats-citypair/output.csv

Task

Provide a meaningful status code and message to handle error at server level