BritishGeologicalSurvey / pyagsapi

pyagsapi - An AGS Utilities API with AGS v4.x Schema Validation & Converter for .ags<-->.xslx files
https://britishgeologicalsurvey.github.io/pyagsapi/
GNU Lesser General Public License v3.0
12 stars 2 forks source link

internal server error response when doing BGS checks & f=JSON #57

Closed KoalaGeo closed 1 year ago

KoalaGeo commented 1 year ago

East West Rail BGS Pre October 2018 upload (partial) .ags

ximenesuk commented 1 year ago

Checking on latest I get this log message:

INFO | 2023-02-22 14:33:11,248 | request | Request: id: afSkF5sF path: http://localhost:8080/validate/
{"loglevel": "info", "workers": 2, "bind": "0.0.0.0:80", "graceful_timeout": 120, "timeout": 120, "keepalive": 5, "errorlog": "-", "accesslog": "-", "workers_per_core": 1.0, "use_max_workers": null, "host": "0.0.0.0", "port": "80"}
  Checking lines...
  Loading tables...
  Checking headings and groups...
  Checking file schema...
/usr/local/lib/python3.11/site-packages/shapely/predicates.py:798: RuntimeWarning: invalid value encountered in intersects
  return lib.intersects(a, b, **kwargs)
ERROR | 2023-02-22 14:33:59,889 | uvicorn.error | Exception in ASGI application

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 94, in receive
    return self.receive_nowait()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 89, in receive_nowait
    raise WouldBlock
anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 78, in call_next
    message = await recv_stream.receive()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/anyio/streams/memory.py", line 114, in receive
    raise EndOfStream
anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 419, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 271, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 118, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 108, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/app/main.py", line 111, in log_requests
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 84, in call_next
    raise app_exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 70, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/usr/local/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/app/routes.py", line 110, in validate
    result = validation.validate(
             ^^^^^^^^^^^^^^^^^^^^
  File "/app/app/validation.py", line 60, in validate
    result = checker(filename, standard_AGS4_dictionary=dictionary_file)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/app/checkers.py", line 76, in check_bgs
    result = func(tables)
             ^^^^^^^^^^^^
  File "/app/app/bgs_rules.py", line 210, in check_loca_within_great_britain
    result = location.apply(check_coordinates, axis=1)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/geopandas/geodataframe.py", line 1482, in apply
    result = super().apply(
             ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pandas/core/frame.py", line 9568, in apply
    return op.apply().__finalize__(self, method="apply")
           ^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pandas/core/apply.py", line 764, in apply
    return self.apply_standard()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pandas/core/apply.py", line 891, in apply_standard
    results, res_index = self.apply_series_generator()
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/pandas/core/apply.py", line 907, in apply_series_generator
    results[i] = self.f(v)
                 ^^^^^^^^^
  File "/app/app/bgs_rules.py", line 189, in check_coordinates
    ni_grid_geometry = Point(to_irish_grid.transform(row['geometry'].x, row['geometry'].y))
                                                     ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/shapely/geometry/point.py", line 88, in x
    return shapely.get_x(self)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/shapely/decorators.py", line 77, in wrapped
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/shapely/_geometry.py", line 271, in get_x
    return lib.get_x(point, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
shapely.errors.GEOSException: UnsupportedOperationException: getX called on empty Point

INFO | 2023-02-22 14:34:00,442 | request | called by 172.17.0.1
KoalaGeo commented 1 year ago

Is it the same error with these 3 - https://github.com/BritishGeologicalSurvey/pyagsapi/wiki/File-Summaries#json-response--internal-server-error-txt-works ??

ximenesuk commented 1 year ago

It looks like there is another issue with this file:

[2023-02-22 15:17:03 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:7)
[2023-02-22 15:17:03 +0000] [1] [WARNING] Worker with pid 7 was terminated due to signal 6
[2023-02-22 15:17:03 +0000] [10] [INFO] Booting worker with pid: 10

It times out and crashes! The file is relatively big, 30 MB, but less than the 50 MB supported. It would be good to islotae the two problems, if there are two.

ximenesuk commented 1 year ago

Is it the same error with these 3 - https://github.com/BritishGeologicalSurvey/pyagsapi/wiki/File-Summaries#json-response--internal-server-error-txt-works ??

These were failing for a different reason which I have now fixed. A bug in the BGS code. I'll use on of those files as a test case.

ETA: there is a test case but the failure was in serialising to JSON and so only apparent in the API. We may need to review API tests for BGS rules.