search

onaio / onadata

Collect, Analyze and Share
https://ona.io
Other
183 stars 133 forks source link

Handle deadlock exceptions during submissions and project edits #908

Open ukanga opened 7 years ago

ukanga commented 7 years ago 
Internal Server Error: /submission

OperationalError at /submission
deadlock detected
DETAIL:  Process 520 waits for ShareLock on transaction 116627497; blocked by process 522.
Process 522 waits for ShareLock on transaction 116627494; blocked by process 520.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (349,32) in relation "logger_project"

Request Method: POST
Request URL: https://odk.ona.io/submission
Django Version: 1.9.5
Python Executable: /..../.virtualenvs/onadata/bin/uwsgi
Python Version: 2.7.6
Python Path: ['/..../.virtualenvs/onadata/src/savreaderwriter/savReaderWriter', '/..../.virtualenvs/onadata/src/savreaderwriter/savReaderWriter/cWriterow', '.', '', '/..../.virtualenvs/onadata/src/django-nose', '/..../.virtualenvs/onadata/src/django-digest', '/..../.virtualenvs/onadata/src/j2xclient', '/..../.virtualenvs/onadata/src/cors-headers', '/..../.virtualenvs/onadata/src/pricing', '/..../.virtualenvs/onadata/src/savreaderwriter', '/..../.virtualenvs/onadata/src/google-export', '/..../.virtualenvs/onadata/lib/python2.7', '/..../.virtualenvs/onadata/lib/python2.7/plat-x86_64-linux-gnu', '/..../.virtualenvs/onadata/lib/python2.7/lib-tk', '/..../.virtualenvs/onadata/lib/python2.7/lib-old', '/..../.virtualenvs/onadata/lib/python2.7/lib-dynload', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/..../.virtualenvs/onadata/local/lib/python2.7/site-packages']
Server time: Wed, 8 Feb 2017 10:07:29 -0500
Installed Applications:
('google_export',
 'pricing',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'django.contrib.admin',
 'django.contrib.admindocs',
 'django.contrib.gis',
 'registration',
 'django_nose',
 'django_digest',
 'corsheaders',
 'oauth2_provider',
 'rest_framework',
 'rest_framework.authtoken',
 'taggit',
 'readonly',
 'onadata.apps.logger',
 'onadata.apps.viewer',
 'onadata.apps.main',
 'onadata.apps.restservice',
 'onadata.apps.api',
 'guardian',
 'djcelery',
 'onadata.apps.sms_support',
 'onadata.libs',
 'reversion')
Installed Middleware:
('multidb.middleware.PinningRouterMiddleware',
 'onadata.libs.profiling.sql.SqlTimingMiddleware',
 'django.middleware.http.ConditionalGetMiddleware',
 'django.middleware.cache.UpdateCacheMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.cache.FetchFromCacheMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'onadata.libs.utils.middleware.LocaleMiddlewareWithTweaks',
 'django.middleware.csrf.CsrfViewMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'onadata.libs.utils.middleware.HTTPResponseNotAllowedMiddleware',
 'readonly.middleware.DatabaseReadOnlyMiddleware')

Traceback:

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  149.                     response = self.process_exception_by_middleware(e, request)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/core/handlers/base.py" in get_response
  147.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/views/decorators/csrf.py" in wrapped_view
  58.         return view_func(*args, **kwargs)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/rest_framework/viewsets.py" in view
  87.             return self.dispatch(request, *args, **kwargs)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
  466.             response = self.handle_exception(exc)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/rest_framework/views.py" in dispatch
  463.             response = handler(request, *args, **kwargs)

File "./onadata/apps/api/viewsets/xform_submission_viewset.py" in create
  149.             error, instance = create_fn(username, request)

File "./onadata/apps/api/viewsets/xform_submission_viewset.py" in create_instance_from_xml
  79.     return safe_create_instance(username, xml_file, media_files, None, request)

File "./onadata/libs/utils/logger_tools.py" in safe_create_instance
  332.             username, xml_file, media_files, uuid=uuid, request=request)

File "./onadata/libs/utils/logger_tools.py" in create_instance
  299.                 date_created_override)

File "./onadata/libs/utils/logger_tools.py" in save_submission
  205.     instance = _get_instance(xml, new_uuid, submitted_by, status, xform)

File "./onadata/libs/utils/logger_tools.py" in _get_instance
  96.             xml=xml, user=submitted_by, status=status, xform=xform)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/models/manager.py" in manager_method
  122.                 return getattr(self.get_queryset(), name)(*args, **kwargs)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/models/query.py" in create
  401.         obj.save(force_insert=True, using=self.db)

File "./onadata/apps/logger/models/instance.py" in save
  455.         super(Instance, self).save(*args, **kwargs)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/models/base.py" in save
  708.                        force_update=force_update, update_fields=update_fields)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/models/base.py" in save_base
  745.                                    update_fields=update_fields, raw=raw, using=using)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py" in send
  192.             response = receiver(signal=self, sender=sender, **named)

File "./onadata/apps/logger/models/instance.py" in post_save_submission
  472.         save_full_json(instance.pk, created)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/celery/app/task.py" in __call__
  420.             return self.run(*args, **kwargs)

File "./onadata/apps/logger/models/instance.py" in save_full_json
  208.             instance.save(update_fields=['json'])

File "./onadata/apps/logger/models/instance.py" in save
  455.         super(Instance, self).save(*args, **kwargs)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/models/base.py" in save
  708.                        force_update=force_update, update_fields=update_fields)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/models/base.py" in save_base
  745.                                    update_fields=update_fields, raw=raw, using=using)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py" in send
  192.             response = receiver(signal=self, sender=sender, **named)

File "./onadata/apps/logger/models/instance.py" in post_save_submission
  473.         update_project_date_modified(instance.pk, created)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/celery/app/task.py" in __call__
  420.             return self.run(*args, **kwargs)

File "./onadata/apps/logger/models/instance.py" in update_project_date_modified
  222.         instance.xform.project.save(update_fields=['date_modified'])

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/models/base.py" in save
  708.                        force_update=force_update, update_fields=update_fields)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/models/base.py" in save_base
  736.             updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/models/base.py" in _save_table
  801.                                       forced_update)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/models/base.py" in _do_update
  851.         return filtered._update(values) > 0

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/models/query.py" in _update
  645.         return query.get_compiler(self.db).execute_sql(CURSOR)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
  1149.         cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
  848.             cursor.execute(sql, params)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/backends/utils.py" in execute
  64.                 return self.cursor.execute(sql, params)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/utils.py" in __exit__
  95.                 six.reraise(dj_exc_type, dj_exc_value, traceback)

File "/..../.virtualenvs/onadata/local/lib/python2.7/site-packages/django/db/backends/utils.py" in execute
  64.                 return self.cursor.execute(sql, params)

Exception Type: OperationalError at /submission
Exception Value: deadlock detected
DETAIL:  Process 520 waits for ShareLock on transaction 116627497; blocked by process 522.
Process 522 waits for ShareLock on transaction 116627494; blocked by process 520.
HINT:  See server log for query details.
CONTEXT:  while updating tuple (349,32) in relation "logger_project"

Request information:
GET: No GET data

POST: No POST data

FILES: No FILES data

COOKIES:
multidb_pin_writes = 'y'
lincmba commented 4 years ago

@ukanga is this still an issue?