eventlet worker class broken on gunicorn 20?

[gforcada]

Hi all!

We have two django projects running with gunicorn and being managed by supervisor.

Up until last week we were using gunicorn[eventlet]==19.9.0 with eventlet==0.25.1 and we bumped gunicorn to gunicorn[eventlet]==20.0.4. Since then we had to switch to the gthread worker class as on every request we were getting the following stacktrace:

[2020-01-28 10:50:40 +0100] [30238] [ERROR] Error handling request /rss.xml
Traceback (most recent call last):
  File "/python3.7/site-packages/gunicorn/workers/base_async.py", line 55, in handle
    self.handle_request(listener_name, req, client, addr)
  File "/python3.7/site-packages/gunicorn/workers/base_async.py", line 106, in handle_request
    respiter = self.wsgi(environ, resp.start_response)
  File "/python3.7/site-packages/sentry_sdk/integrations/django/__init__.py", line 108, in sentry_patched_wsgi_handler
    return SentryWsgiMiddleware(bound_old_app)(environ, start_response)
  File "/python3.7/site-packages/sentry_sdk/integrations/wsgi.py", line 120, in __call__
    reraise(*_capture_exception(hub))
  File "/python3.7/site-packages/sentry_sdk/_compat.py", line 57, in reraise
    raise value
  File "/python3.7/site-packages/sentry_sdk/integrations/wsgi.py", line 116, in __call__
    _sentry_start_response, start_response, span
  File "/python3.7/site-packages/django/core/handlers/wsgi.py", line 139, in __call__
    signals.request_started.send(sender=self.__class__, environ=environ)
  File "/python3.7/site-packages/django/dispatch/dispatcher.py", line 175, in send
    for receiver in self._live_receivers(sender)
  File "/python3.7/site-packages/django/dispatch/dispatcher.py", line 175, in <listcomp>
    for receiver in self._live_receivers(sender)
  File "/python3.7/site-packages/django/db/__init__.py", line 57, in close_old_connections
    conn.close_if_unusable_or_obsolete()
  File "/python3.7/site-packages/django/db/backends/base/base.py", line 514, in close_if_unusable_or_obsolete
    self.close()
  File "/python3.7/site-packages/django/db/backends/base/base.py", line 279, in close
    self.validate_thread_sharing()
  File "/python3.7/site-packages/django/db/backends/base/base.py", line 547, in validate_thread_sharing
    % (self.alias, self._thread_ident, _thread.get_ident())
django.db.utils.DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread. The object with alias 'default' was created in thread id 140283393784016 and this is thread id 140283342273728.

I'm reporting it here rather than on django as just changing the worker class fixed it. Our servers seem happy with the gthread worker class, so we are not in a hurry to get this sorted out.

Still, the question remains, of is there is something broken with this combination? So far I could not find anything here on gunicorn nor in eventlet issue trackers :confused:

TIA!

[jamadden]

My first guess would be issues with the way eventlet is monkey-patching. Are you preloading your app? Can you post your gunicorn configuration? Does it work with the gevent worker?

[gforcada]

@jamadden thanks for the quick answering!

As for gevent we did not try it, I gave it a quick go now and I get the exact same traceback.

As for preloading, no, we were not doing it, the gunicorn config is the following:

import sentry_sdk
from sentry_sdk.integrations.django import DjangoIntegration

sentry_sdk.init(
    dsn='XXXX',
    environment='staging',
    integrations=[DjangoIntegration()],
)

bind = ['127.0.0.1:8000',]

worker_class = 'gthread'
workers = 1
if worker_class == 'gthread':
    threads = 2

logconfig = 'logging.conf'
raw_env = [
    'DJANGO_DEPLOYMENT_ENVIRONMENT=XXX',
    'DJANGO_SETTINGS_MODULE=XXX',
]

[jamadden]

Django use a module-scoped thread local object to track connections. This thread-local is allocated when django.db is first imported. If that happens before monkey-patching, it's not going to work with greenlets — it'll track all connections that are opened across the entire application, instead of just the current greenlet (posing as a "thread" when monkey-patched).

I would guess that something, probably the import of DjangoIntegration in your configuration, is resulting in django.db being imported. Because configuration is imported before picking a worker class, it happens before monkey-patching too. You could verify this by examining/printing sys.modules at the end of your configuration file.

You can try putting from gevent import monkey; monkey.patch_all() at the very top of your configuration (first confirm that django.db hasn't been imported yet!). Or try the custom entry point trick described in another issue in this repo.

[gforcada]

Thanks we will have a look at that :+1:

[webjunkie]

Just a note here to possibly help someone: We experienced similar issues when trying to upgrade Django from 2 to 3. Also see here https://github.com/celery/celery/issues/5924

[Achilles718611]

I faced with the same problem. I think it's not Gunicorn problem. It seems related with Django 3 and Celery.

[piasekps]

I have the same error: we are using python 3.7 Django 1.11 and Celery 4.3.0

Did you guys solve the issue? Can you give me some hint on how to solve it?

[jessequinn]

we have the same issue with gunicorn, and otel instrumentation. Any ideas?