search

Linaro / squad

Software Quality Dashboard
GNU General Public License v3.0
59 stars 41 forks source link

squad.core.tasks.notification.notification_timeout raises IntegrityError #363

Closed mwasilew closed 5 years ago

mwasilew commented 5 years ago 
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]: [2018-10-03 08:16:54,099: ERROR/ForkPoolWorker-2] Task squad.core.tasks.notification.notification_timeout[371bb6a2-1286-41a3-aefe-a6fd88b730bf] raised unexpected: IntegrityError('duplicate key value violates unique constraint
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]: Traceback (most recent call last):
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/query.py", line 464, in get_or_create
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     return self.get(**lookup), False
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/query.py", line 380, in get
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     self.model._meta.object_name
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]: squad.core.models.DoesNotExist: NotificationDelivery matching query does not exist.
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]: During handling of the above exception, another exception occurred:
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]: Traceback (most recent call last):
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     return self.cursor.execute(sql, params)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]: psycopg2.IntegrityError: duplicate key value violates unique constraint "core_notificationdelivery_status_id_key"
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]: DETAIL:  Key (status_id)=(9919) already exists.
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]: The above exception was the direct cause of the following exception:
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]: Traceback (most recent call last):
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/celery/app/trace.py", line 374, in trace_task
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     R = retval = fun(*args, **kwargs)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/celery/app/trace.py", line 629, in __protected_call__
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     return self.run(*args, **kwargs)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/squad/core/tasks/notification.py", line 49, in notification_timeout
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     send_status_notification(projectstatus)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/squad/core/notification.py", line 194, in send_status_notification
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     send_admin_notification(status, project)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/squad/core/notification.py", line 238, in send_admin_notification
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     if NotificationDelivery.exists(status, subject, txt, html):
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/squad/core/models.py", line 847, in exists
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     html=html_hash,
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     return getattr(self.get_queryset(), name)(*args, **kwargs)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/query.py", line 466, in get_or_create
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     return self._create_object_from_params(lookup, params)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/query.py", line 506, in _create_object_from_params
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     six.reraise(*exc_info)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/utils/six.py", line 686, in reraise
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     raise value
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/query.py", line 498, in _create_object_from_params
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     obj = self.create(**params)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/query.py", line 394, in create
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     obj.save(force_insert=True, using=self.db)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/base.py", line 808, in save
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     force_update=force_update, update_fields=update_fields)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/base.py", line 838, in save_base
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/base.py", line 924, in _save_table
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/base.py", line 963, in _do_insert
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     using=using, raw=raw)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/manager.py", line 85, in manager_method
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     return getattr(self.get_queryset(), name)(*args, **kwargs)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/query.py", line 1076, in _insert
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     return query.get_compiler(using=using).execute_sql(return_id)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/models/sql/compiler.py", line 1113, in execute_sql
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     cursor.execute(sql, params)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     return self.cursor.execute(sql, params)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/utils.py", line 94, in __exit__
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     six.reraise(dj_exc_type, dj_exc_value, traceback)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/utils/six.py", line 685, in reraise
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     raise value.with_traceback(tb)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:   File "/srv/qa-reports.linaro.org/lib/python3.5/site-packages/django/db/backends/utils.py", line 64, in execute
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]:     return self.cursor.execute(sql, params)
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]: django.db.utils.IntegrityError: duplicate key value violates unique constraint "core_notificationdelivery_status_id_key"
Oct 03 08:16:54 qa-reports-worker-1 celery[28075]: DETAIL:  Key (status_id)=(9919) already exists.
mwasilew commented 5 years ago

I think we're hitting race condition as described here: http://www.ramandv.com/blog/django-get_or_create-integrityerror-due-to-racing-condition/