search

yougov / velociraptor

BSD 3-Clause "New" or "Revised" License
11 stars 1 forks source link

Django not closing DB connections #181

Open ghost opened 8 years ago

ghost commented 8 years ago

Originally reported by: Lorenzo Bolla (Bitbucket: lbolla, GitHub: lbolla)

Django does not seem to be closing connections to PG. Eventually, max_connections limit is hit and no more connections are accepted, causing the below error.

#!python

ERROR:django.request.tastypie:Internal Server Error: /api/v1/apps/cu_api/
Traceback (most recent call last):
  File "/app/.heroku/venv/lib/python2.7/site-packages/tastypie/resources.py", line 202, in wrapper
    response = callback(request, *args, **kwargs)
  File "/app/.heroku/venv/lib/python2.7/site-packages/tastypie/resources.py", line 442, in dispatch_detail
    return self.dispatch('detail', request, **kwargs)
  File "/app/.heroku/venv/lib/python2.7/site-packages/tastypie/resources.py", line 460, in dispatch
    self.is_authenticated(request)
  File "/app/.heroku/venv/lib/python2.7/site-packages/tastypie/resources.py", line 543, in is_authenticated
    auth_result = self._meta.authentication.is_authenticated(request)
  File "/app/.heroku/venv/lib/python2.7/site-packages/tastypie/authentication.py", line 500, in is_authenticated
    check = backend.is_authenticated(request, **kwargs)
  File "/app/.heroku/venv/lib/python2.7/site-packages/tastypie/authentication.py", line 251, in is_authenticated
    return request.user.is_authenticated()
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/utils/functional.py", line 225, in inner
    self._setup()
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/utils/functional.py", line 365, in _setup
    self._wrapped = self._setupfunc()
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/contrib/auth/middleware.py", line 22, in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/contrib/auth/middleware.py", line 10, in get_user
    request._cached_user = auth.get_user(request)
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/contrib/auth/__init__.py", line 174, in get_user
    user = backend.get_user(user_id)
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/contrib/auth/backends.py", line 93, in get_user
    return UserModel._default_manager.get(pk=user_id)
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/models/query.py", line 328, in get
    num = len(clone)
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/models/query.py", line 144, in __len__
    self._fetch_all()
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/models/query.py", line 965, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/models/query.py", line 238, in iterator
    results = compiler.execute_sql()
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 838, in execute_sql
    cursor = self.connection.cursor()
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 164, in cursor
    cursor = self.make_cursor(self._cursor())
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 135, in _cursor
    self.ensure_connection()
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 130, in ensure_connection
    self.connect()
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 130, in ensure_connection
    self.connect()
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 119, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/app/.heroku/venv/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 176, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/app/.heroku/venv/lib/python2.7/site-packages/psycopg2/__init__.py", line 178, in connect
    return _connect(dsn, connection_factory=connection_factory, async=async)
  File "/app/.heroku/venv/lib/python2.7/site-packages/psycogreen/gevent.py", line 29, in gevent_wait_callback
    state = conn.poll()
OperationalError: asynchronous connection failed
ghost commented 8 years ago

Original comment by Lorenzo Bolla (Bitbucket: lbolla, GitHub: lbolla):

One option to make sure connections are cleaned up is to use gunicorn's --max-requests so that GUnicorn workers are killed, and connection freed, after few requests. Another better option is to use a proper connection pool.