search

jneight / django-db-geventpool

Another DB pool using gevent
Apache License 2.0
167 stars 29 forks source link

Django Async: Exception raised when many connections are opened #90

Open JannKleen opened 2 months ago

JannKleen commented 2 months ago

I got the following traceback while I was testing that the maximum pool size didn't impact our database. I assume I exceeded the max pool size here?

We're using Django + ASGI + Django-channels + uvicorn, but this happened in sync views so far. Do you have any ideas what could be causing this? I'm happy to help debugging this further.

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 518, in thread_handler
    raise exc_info[1]
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 42, in inner
    response = await get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 518, in thread_handler
    raise exc_info[1]
  File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
    response = await wrapped_callback(
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 468, in __call__
    ret = await asyncio.shield(exec_coro)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asgiref/current_thread_executor.py", line 40, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 522, in thread_handler
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/views.py", line 84, in sentry_wrapped_callback
    return callback(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/views/decorators/csrf.py", line 65, in _view_wrapper
    return view_func(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/viewsets.py", line 124, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 497, in dispatch
    self.initial(request, *args, **kwargs)
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/__init__.py", line 304, in sentry_patched_drf_initial
    return old_drf_initial(self, request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 414, in initial
    self.perform_authentication(request)
  File "/usr/local/lib/python3.11/site-packages/rest_framework/views.py", line 324, in perform_authentication
    request.user
  File "/usr/local/lib/python3.11/site-packages/rest_framework/request.py", line 231, in user
    self._authenticate()
  File "/usr/local/lib/python3.11/site-packages/rest_framework/request.py", line 384, in _authenticate
    user_auth_tuple = authenticator.authenticate(self)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/knox/auth.py", line 53, in authenticate
    user, auth_token = self.authenticate_credentials(auth[1])
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/knox/auth.py", line 65, in authenticate_credentials
    for auth_token in AuthToken.objects.filter(
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 400, in __iter__
    self._fetch_all()
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1928, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
    results = compiler.execute_sql(
              ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1560, in execute_sql
    cursor = self.connection.cursor()
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 316, in cursor
    return self._cursor()
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 292, in _cursor
    self.ensure_connection()
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 275, in ensure_connection
    self.connect()
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/utils.py", line 1711, in runner
    return sentry_patched_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/__init__.py", line 659, in connect
    return real_connect(self)
           ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/utils/asyncio.py", line 26, in inner
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 256, in connect
    self.connection = self.get_new_connection(conn_params)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django_db_geventpool/backends/base.py", line 41, in get_new_connection
    self.connection = self.pool.get()
                      ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/django_db_geventpool/backends/pool.py", line 38, in get
    conn = self.pool.get()
           ^^^^^^^^^^^^^^^
  File "src/gevent/queue.py", line 335, in gevent._gevent_cqueue.Queue.get
  File "src/gevent/queue.py", line 350, in gevent._gevent_cqueue.Queue.get
  File "src/gevent/queue.py", line 327, in gevent._gevent_cqueue.Queue._Queue__get_or_peek
  File "src/gevent/_waiter.py", line 154, in gevent._gevent_c_waiter.Waiter.get
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 61, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_greenlet_primitives.py", line 65, in gevent._gevent_c_greenlet_primitives.SwitchOutGreenletWithLoop.switch
  File "src/gevent/_gevent_c_greenlet_primitives.pxd", line 35, in gevent._gevent_c_greenlet_primitives._greenlet_switch
gevent.exceptions.LoopExit: This operation would block forever
    Hub: <Hub '' at 0x7f839a9ed3f0 epoll pending=0 ref=0 fileno=79 thread_ident=0x7f83947fd700>
    Handles:
[]
SamkeetJain commented 1 month ago

+1 Facing same issue. Did anyone find a workaround?

JannKleen commented 1 month ago

We decided to use pgbouncer for now.