pin mangum to 0.12

[vincentsarago]

{
  "detail": "the pool 'pool-5' is already closed"
}

[vincentsarago]

note: I can't reproduce the error using docker compose so it might only be in AWS Lambda

Full log 👇

[ERROR] 2022-02-25T18:39:10.100Z e5ed855f-c427-40c7-bfad-ed5157bc2d31 Exception in 'http' protocol. Traceback (most recent call last): 
File "/tmp/pip-target-4wdyrqzh/lib/python/anyio/streams/memory.py", line 81, in receive 
File "/tmp/pip-target-4wdyrqzh/lib/python/anyio/streams/memory.py", line 76, in receive_nowait anyio.WouldBlock During handling of the above exception, another exception occurred: Traceback (most recent call last): 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/middleware/base.py", line 41, in call_next 
File "/tmp/pip-target-4wdyrqzh/lib/python/anyio/streams/memory.py", line 101, in receive anyio.EndOfStream During handling of the above exception, another exception occurred: Traceback (most recent call last): 
File "/tmp/pip-target-4wdyrqzh/lib/python/mangum/protocols/http.py", line 79, in run 
File "/tmp/pip-target-4wdyrqzh/lib/python/fastapi/applications.py", line 259, in __call__ 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/applications.py", line 112, in __call__ 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/middleware/errors.py", line 181, in __call__ 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/middleware/errors.py", line 159, in __call__ 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette_cramjam/middleware.py", line 52, in __call__ 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette_cramjam/middleware.py", line 101, in __call__ 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/middleware/base.py", line 63, in __call__ 
File "/tmp/pip-target-4wdyrqzh/lib/python/titiler/core/middleware.py", line 38, in dispatch 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/middleware/base.py", line 44, in call_next 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/middleware/base.py", line 34, in coro 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/middleware/cors.py", line 84, in __call__ 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/exceptions.py", line 82, in __call__ 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/exceptions.py", line 71, in __call__ 
File "/tmp/pip-target-4wdyrqzh/lib/python/fastapi/middleware/asyncexitstack.py", line 21, in __call__ 
File "/tmp/pip-target-4wdyrqzh/lib/python/fastapi/middleware/asyncexitstack.py", line 18, in __call__ 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/routing.py", line 656, in __call__ 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/routing.py", line 259, in handle 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/routing.py", line 61, in app 
File "/tmp/pip-target-4wdyrqzh/lib/python/fastapi/routing.py", line 227, in app 
File "/tmp/pip-target-4wdyrqzh/lib/python/fastapi/routing.py", line 162, in run_endpoint_function 
File "/tmp/pip-target-4wdyrqzh/lib/python/starlette/concurrency.py", line 39, in run_in_threadpool 
File "/tmp/pip-target-4wdyrqzh/lib/python/anyio/to_thread.py", line 28, in run_sync 
File "/tmp/pip-target-4wdyrqzh/lib/python/anyio/_backends/_asyncio.py", line 818, in run_sync_in_worker_thread 
File "/tmp/pip-target-4wdyrqzh/lib/python/anyio/_backends/_asyncio.py", line 754, in run 
File "/tmp/pip-target-4wdyrqzh/lib/python/titiler/pgstac/factory.py", line 148, in tile 
File "/tmp/pip-target-4wdyrqzh/lib/python/titiler/pgstac/mosaic.py", line 238, in tile 
File "/tmp/pip-target-4wdyrqzh/lib/python/titiler/pgstac/mosaic.py", line 154, in assets_for_tile 
File "/tmp/pip-target-4wdyrqzh/lib/python/cachetools/__init__.py", line 642, in wrapper 
File "/tmp/pip-target-4wdyrqzh/lib/python/titiler/pgstac/mosaic.py", line 201, in get_assets 
File "/var/lang/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) 
File "/tmp/pip-target-4wdyrqzh/lib/python/psycopg_pool/pool.py", line 114, in connection 
File "/tmp/pip-target-4wdyrqzh/lib/python/psycopg_pool/pool.py", line 140, in getconn 
File "/tmp/pip-target-4wdyrqzh/lib/python/psycopg_pool/base.py", line 141, in _check_open_getconn psycopg_pool.PoolClosed: the pool 'pool-1' is already closed

ile "/tmp/pip-target-4wdyrqzh/lib/python/anyio/streams/memory.py", line 81, in receive 
File "/tmp/pip-target-4wdyrqzh/lib/python/anyio/streams/memory.py", line 76, in receive_nowait anyio.WouldBlock During handling of the above exception, another exception occurred: Traceback (most recent call last): 

I'm quite surprise that we are using anyio

edit: ☝️

Starlette and FastAPI are based on AnyIO, which makes them compatible with both Python's standard library asyncio and Trio.

that is actually the default

cc @sharkinsspatial

[vincentsarago]

there was a new release of mangum 3days ago https://github.com/jordaneremieff/mangum/releases/tag/0.13.0

maybe they've changed something

[vincentsarago]

Confirmed, pinning mangum to 0.12.3 fixes the issue

[vincentsarago]

@jordaneremieff sorry for tagging you here, but there is a strange behaviour with the newly release mangum==0.13.0

I'm not sure what's going on. I'll try to dig deeper tomorrow. Right now I'm pinning to 0.12.3 but I'll try 0.12.4 to narrow things.

Our lambda handler look like 👇

https://github.com/developmentseed/eoAPI/blob/66085ac249752c6ef0adbb09026f187fe21f29f6/deployment/handlers/raster_handler.py#L1-L12

To Do:

• deploy with 0.12.4
• read release change log of mangum to see if we did miss any breaking change 😄
• write re-productible simple application

[jordaneremieff]

Hi @vincentsarago, the latest release involved quite a bit of refactoring to remove WebSocket support and start preparing for a larger refactor of the handlers generally. Not sure what change may be specifically related here.

Could you confirm what happens if you configure handler = Mangum(app, lifespan="off") instead of lifespan="auto"? Also, once you've investigated and are able to reproduce, could you open an issue on the Mangum repo?

[vincentsarago]

"""AWS Lambda handler."""

from mangum import Mangum
from starlette.requests import Request
from fastapi import FastAPI

app = FastAPI()
@app.get("/yo")
async def ping(request: Request):
    """Health check."""
    return request.app.state.v

@app.on_event("startup")
async def startup_event() -> None:
    """Connect to database on startup."""
    app.state.v = "In"

@app.on_event("shutdown")
async def shutdown_event() -> None:
    """Close database connection."""
    app.state.v = "Out"

handler = Mangum(app, lifespan="auto")

☝️ this simple app return In for mangum=0.12.4 and Out for mangum=0.13.

When setting lifespan="off" I'm getting Internal Server Error

[ERROR] 2022-03-01T22:13:19.759Z    45e46869-c976-4f32-bd3d-cab179cf8caf    Exception in 'http' protocol.
Traceback (most recent call last):
  File "/var/task/starlette/datastructures.py", line 661, in __getattr__
    return self._state[key]
KeyError: 'v'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/var/task/mangum/protocols/http.py", line 81, in run
    await app(self.request.scope, self.receive, self.send)
  File "/var/task/fastapi/applications.py", line 259, in __call__
    await super().__call__(scope, receive, send)
  File "/var/task/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/var/task/starlette/middleware/errors.py", line 181, in __call__
    raise exc
  File "/var/task/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/var/task/starlette/exceptions.py", line 82, in __call__
    raise exc
  File "/var/task/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/var/task/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/var/task/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "/var/task/starlette/routing.py", line 656, in __call__
    await route.handle(scope, receive, send)
  File "/var/task/starlette/routing.py", line 259, in handle
    await self.app(scope, receive, send)
  File "/var/task/starlette/routing.py", line 61, in app
    response = await func(request)
  File "/var/task/fastapi/routing.py", line 227, in app
    raw_response = await run_endpoint_function(
  File "/var/task/fastapi/routing.py", line 160, in run_endpoint_function
    return await dependant.call(**values)
  File "/var/task/handler.py", line 16, in ping
    return request.app.state.v
  File "/var/task/starlette/datastructures.py", line 664, in __getattr__
    raise AttributeError(message.format(self.__class__.__name__, key))
AttributeError: 'State' object has no attribute 'v'

When setting lifespan="on", I'm seeing the same result as with auto

[jordaneremieff]

@vincentsarago I've released a new version 0.14.0 that should include the fix for this.

[vincentsarago]

Thanks a ton @jordaneremieff 🙏