search

stac-utils / pgstac

Schema, functions and a python library for storing and accessing STAC collections and items in PostgreSQL
MIT License
153 stars 39 forks source link

PgStac no longer works with Azure Flexible Postgres server #227

Closed martys closed 1 year ago

martys commented 1 year ago

A few days ago Microsoft has bumped up a minor version of a "Azure Database for PostgreSQL flexible server" to version 13.12

After that all PGStac methods using that are utilized by Stac-Fast-Api stopped working ! resulting in a "segmentation fault" and not returning any data back.

Executing the same queries or stored procedures using a PGAdmin works fine, indicating a problem with pgstab library

We are in the middle of production outage right now due to this, and will add more details as they become available.

martys commented 1 year ago

here's some log's from the python code 

Traceback (most recent call last):

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/pool.py", line 218, in release

    self._con.terminate()

AttributeError: 'NoneType' object has no attribute 'terminate'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 376, in run_asgi

    result = await app(self.scope, self.receive, self.send)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in __call__

    return await self.app(scope, receive, send)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/fastapi/applications.py", line 261, in __call__

    await super().__call__(scope, receive, send)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/applications.py", line 112, in __call__

    await self.middleware_stack(scope, receive, send)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 181, in __call__

    raise exc

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/middleware/errors.py", line 159, in __call__

    await self.app(scope, receive, _send)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/stac_fastapi/api/middleware.py", line 76, in __call__

    await self.app(scope, receive, send)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/middleware/cors.py", line 84, in __call__

    await self.app(scope, receive, send)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/brotli_asgi/__init__.py", line 81, in __call__

    await br_responder(scope, receive, send)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/brotli_asgi/__init__.py", line 126, in __call__

    await self.app(scope, receive, self.send_with_brotli)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/exceptions.py", line 82, in __call__

    raise exc

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/exceptions.py", line 71, in __call__

    await self.app(scope, receive, sender)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__

    raise e

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__

    await self.app(scope, receive, send)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/routing.py", line 656, in __call__

    await route.handle(scope, receive, send)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/routing.py", line 259, in handle

    await self.app(scope, receive, send)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/starlette/routing.py", line 61, in app

    response = await func(request)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/fastapi/routing.py", line 227, in app

    raw_response = await run_endpoint_function(

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/fastapi/routing.py", line 160, in run_endpoint_function

    return await dependant.call(**values)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/stac_fastapi/api/routes.py", line 57, in _endpoint

    await func(request=request, **request_data.kwargs()), response_class

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/stac_fastapi/pgstac/core.py", line 42, in all_collections

    async with pool.acquire() as conn:

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/pool.py", line 1012, in __aexit__

    await self.pool.release(con)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/pool.py", line 886, in release

    return await asyncio.shield(ch.release(timeout))

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/pool.py", line 220, in release

    raise ex

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/pool.py", line 210, in release

    await self._con.reset(timeout=budget)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/connection.py", line 1366, in reset

    await self.execute(reset_query, timeout=timeout)

  File "/home/mantasmantas/projects/Atlas-Platform/app/atlas_ui/stac_fastapi/venv/lib/python3.10/site-packages/asyncpg/connection.py", line 317, in execute

    return await self._protocol.query(query, timeout)

  File "asyncpg/protocol/protocol.pyx", line 338, in query

asyncpg.exceptions.ConnectionDoesNotExistError: connection was closed in the middle of operation
 like 1
martys commented 1 year ago

same issue with v14 or v15 as well

bitner commented 1 year ago

TRACKING FOR THIS ISSUE at https://github.com/stac-utils/stac-fastapi-pgstac/issues/65