BeanieODM / beanie

Asynchronous Python ODM for MongoDB
http://beanie-odm.dev/
Apache License 2.0
2.09k stars 219 forks source link

[BUG] get_motor_collection() returning None under volume only #934

Closed ltieman closed 4 months ago

ltieman commented 6 months ago

Describe the bug We are getting back None from get_motor_collection() but only sporadically when we we are under moderate volume.

This is our singleton for init_beanie

beanie_init: bool = False
class MongoSingleton:
    client: AsyncIOMotorClientSession = AsyncIOMotorClient(settings.MONGO_URI, maxPoolSize=400, minPoolSize=50)

    async def startup(self):
        global beanie_init
        if not beanie_init:
            from libraries.mongo.models import models as all_models
            global all_models
            for model in all_models:
                model.model_rebuild(force=True)
            await init_beanie(self.client.db, document_models=all_models)

            beanie_init = True

mongo_singleton = MongoSingleton() 
@asynccontextmanager
async def startup_event(app: FastAPI):
    from libraries.mongo.mongodb import mongo_singleton
    await mongo_singleton.startup()
    yield

fastapi_app = FastAPI(lifespan=startup_event) 

and this is our stacktrace:

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 426, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 84, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/fastapi/applications.py", line 1106, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/sentry_sdk/integrations/starlette.py", line 364, in _sentry_patched_asgi_app
    return await middleware(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/sentry_sdk/integrations/asgi.py", line 146, in _run_asgi3
    return await self._run_app(scope, receive, send, asgi_version=3)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/sentry_sdk/integrations/asgi.py", line 241, in _run_app
    raise exc from None
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/sentry_sdk/integrations/asgi.py", line 234, in _run_app
    return await self.app(
           ^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/starlette/applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/sentry_sdk/integrations/starlette.py", line 157, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/starlette/middleware/cors.py", line 91, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/starlette/middleware/cors.py", line 146, in simple_response
    await self.app(scope, receive, send)
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/sentry_sdk/integrations/starlette.py", line 256, in _sentry_exceptionmiddleware_call
    await old_call(self, scope, receive, send)
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/sentry_sdk/integrations/starlette.py", line 157, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/sentry_sdk/integrations/starlette.py", line 157, in _create_span_call
    return await old_call(app, scope, new_receive, new_send, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 20, in __call__
    raise e
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/fastapi/middleware/asyncexitstack.py", line 17, in __call__
    await self.app(scope, receive, send)
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/starlette/routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/starlette/routing.py", line 66, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/fastapi/routing.py", line 274, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/libraries/api_resources/resource_type.py", line 347, in get_by_id
    return await self.get_by_id_crud(id=id, engine=engine, user=user)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/libraries/api_resources/mongo_resource.py", line 286, in get_by_id_crud
    data = await self.model.find_one(*new_query, fetch_links=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/beanie/odm/queries/find.py", line 1042, in __await__
    document = yield from self._find_one().__await__()  # type: ignore
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/beanie/odm/queries/find.py", line 998, in _find_one
    return await self.document_model.find_many(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/beanie/odm/queries/find.py", line 701, in first_or_none
    res = await self.limit(1).to_list()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/beanie/odm/queries/cursor.py", line 72, in to_list
    cursor = self.motor_cursor
             ^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/beanie/odm/queries/find.py", line 674, in motor_cursor
    ] = self.build_aggregation_pipeline()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/beanie/odm/queries/find.py", line 623, in build_aggregation_pipeline
    ] = construct_lookup_queries(
        ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/beanie/odm/utils/find.py", line 29, in construct_lookup_queries
    construct_query(
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/beanie/odm/utils/find.py", line 56, in construct_query
    raise e
  File "/Users/developer/projects/project/monorepo/python_services/venv/lib/python3.11/site-packages/beanie/odm/utils/find.py", line 54, in construct_query
    link_info.document_class.get_motor_collection().name
AttributeError: 'NoneType' object has no attribute 'name'

Expected behavior We expect this to be able to fetch the links that are coming off of this Document, yet it can not because it is getting back a None. We do not have this behavior happening unless it is under load. When running tests, or under calls that are one at a time, it works. This seems similar to #440 though in our case, it works unless it is under a moderate load (10-20 calls) and only happens sporatically.

ltieman commented 6 months ago

@roman-right we did figure out a workaround by overwriting the get_motor_collection method thusly:

    @classmethod
    def get_motor_collection(cls) -> AsyncIOMotorCollection:
        item = super().get_motor_collection()
        if item is None:
            from libraries.mongo.mongodb import mongo_singleton
            from motor.motor_asyncio import AsyncIOMotorCollection
            cls.set_database(mongo_singleton.client.db)
            name = cls.get_collection_name()
            if not name:
                name = cls.__name__
            cls.set_collection(AsyncIOMotorCollection(
                database=mongo_singleton.client.db,
                name=name,
            ))
            return cls.get_motor_collection()
        return item

We are concerned about the overall impact of this, but it does seem to be working as of now. Any thoughts on why this could be happening?

github-actions[bot] commented 5 months ago

This issue is stale because it has been open 30 days with no activity.

github-actions[bot] commented 4 months ago

This issue was closed because it has been stalled for 14 days with no activity.

treylade commented 3 weeks ago

@ltieman I am now also using your workaround, the get_motor_collection overwrite to tackle my issue. In a multi-tenant Flask context, I have to call init_beanie on every Flask request to switch the tenant database, which leads to get_motor_collection to sporadically return None because of timing issues as described here.

Did you face any issues in the meanwhile with that workaround?

@roman-right is there any update on a fix within Beanie/Bunnet for that issue?