MIT-LCP / physionet-build

The new PhysioNet platform.
https://physionet.org/
BSD 3-Clause "New" or "Revised" License
56 stars 20 forks source link

Unable to change references during copy-edit stage - reference deletion error and server 500 error #1105

Closed li-lcp closed 1 year ago

li-lcp commented 4 years ago

During copy edit stage, the following errors occurred:

(1) deleting 1 reference by pressing on the "x" sign, surprisingly caused two references to be deleted. (The particular sequence of actions that led to this error: adding 3 references at the bottom, and then attempting to delete the first reference on top, caused two references to disappear.) (2) Changing/swapping the content of the references, and then press "Save Content" apparently lead to Server 500 errors persistently.

(My original intention was simply to change the order of the references to match the order in which they appear in the main text.)

tompollard commented 4 years ago

Thanks for raising this @li-lcp. I have added the traceback at the end of this post.

We need to look into point 1. Not sure what happened with that. I have added a note on point 2 below.

(2) Changing/swapping the content of the references, and then press "Save Content" apparently lead to Server 500 errors persistently.

It looks like your changes are raising a duplicate key error, and I think this is an issue with the way the system is setup. I've not explored, but I think the problem is:

1) the database contains a list of references. each reference description must be unique within the project. 2) you are viewing the references in the form, then editing them (e.g. reordering). 3) when you click "save/submit" on the form, the system attempts to save the changes in the database 4) an integrity error is raised because the references that you are saving already exist in the database.

For now, you should be able to bypass the issue by making sure that new references are not an identical match to references that already in the system (e.g. add some temporary text when you save, then remove this temporary text in a later save step if necessary). In the longer term, the system for managing references should be improved.

Internal Server Error: /console/submitted-projects/M1X2WSSvkIOzXJd78Oho/copyedit/

IntegrityError at /console/submitted-projects/M1X2WSSvkIOzXJd78Oho/copyedit/
duplicate key value violates unique constraint "project_reference_description_content_type_8b53e7b5_uniq"
DETAIL:  Key (description, content_type_id, object_id)=(Epilepsy Foundations, Last accessed on July 17th, 2020. https://www.epilepsy.com/, 11, 51) already exists.

Request Method: POST
Request URL: https://www.physionet.org/console/submitted-projects/M1X2WSSvkIOzXJd78Oho/copyedit/

Traceback:

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/db/backends/utils.py" in _execute
 84.                 return self.cursor.execute(sql, params)

The above exception (duplicate key value violates unique constraint "project_reference_description_content_type_8b53e7b5_uniq"
DETAIL:  Key (description, content_type_id, object_id)=(Epilepsy Foundations, Last accessed on July 17th, 2020. https://www.epilepsy.com/, 11, 51) already exists.
) was the direct cause of the following exception:

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/core/handlers/exception.py" in inner
 34.             response = get_response(request)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/core/handlers/base.py" in _get_response
 115.                 response = self.process_exception_by_middleware(e, request)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/core/handlers/base.py" in _get_response
 113.                 response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/contrib/auth/decorators.py" in _wrapped_view
 21.                 return view_func(request, *args, **kwargs)

File "./console/views.py" in handling_view
 84.                 return base_view(request, *args, **kwargs)

File "./console/views.py" in copyedit_submission
 368.                 reference_formset.save()

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/forms/models.py" in save
 669.         return self.save_existing_objects(commit) + self.save_new_objects(commit)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/forms/models.py" in save_existing_objects
 793.                 saved_instances.append(self.save_existing(form, obj, commit=commit))

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/forms/models.py" in save_existing
 650.         return form.save(commit=commit)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/forms/models.py" in save
 458.             self.instance.save()

File "./project/models.py" in save
 383.         super().save(*args, **kwargs)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/db/models/base.py" in save
 741.                        force_update=force_update, update_fields=update_fields)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/db/models/base.py" in save_base
 779.                 force_update, using, update_fields,

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/db/models/base.py" in _save_table
 851.                                       forced_update)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/db/models/base.py" in _do_update
 900.         return filtered._update(values) > 0

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/db/models/query.py" in _update
 760.         return query.get_compiler(self.db).execute_sql(CURSOR)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
 1469.         cursor = super().execute_sql(result_type)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/db/models/sql/compiler.py" in execute_sql
 1140.             cursor.execute(sql, params)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/db/backends/utils.py" in execute
 67.         return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/db/backends/utils.py" in _execute_with_wrappers
 76.         return executor(sql, params, many, context)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/db/backends/utils.py" in _execute
 84.                 return self.cursor.execute(sql, params)

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/db/utils.py" in __exit__
 89.                 raise dj_exc_value.with_traceback(traceback) from exc_value

File "/physionet/python-env/physionet/lib/python3.7/site-packages/django/db/backends/utils.py" in _execute
 84.                 return self.cursor.execute(sql, params)

Exception Type: IntegrityError at /console/submitted-projects/M1X2WSSvkIOzXJd78Oho/copyedit/
Exception Value: duplicate key value violates unique constraint "project_reference_description_content_type_8b53e7b5_uniq"
DETAIL:  Key (description, content_type_id, object_id)=(Epilepsy Foundations, Last accessed on July 17th, 2020. https://www.epilepsy.com/, 11, 51) already exists.

Settings:
Using settings module physionet.settings.production
li-lcp commented 4 years ago

Thanks! It's very helpful to know that the error was due to the fact that the system thought that there were duplicates in the references, when in fact I have just switched the order in which they appeared, and all of them were unique.

I am guessing what happened was that the system was taking each reference in the updated list and checking against the references from the previous saved session for duplicate.

I finally got around the problem by:

That appeared to have worked!

tompollard commented 4 years ago

Thanks @li-lcp I'm glad the temporary fix worked. We'll keep the issue open, so that the underlying bug can fixed at some point.

tompollard commented 1 year ago

As noted in https://github.com/MIT-LCP/physionet-build/issues/1557, this should now be fixed (thanks @amitupreti!). References now have an order and this order is used to sort references.

tompollard commented 1 year ago

Fixed in https://github.com/MIT-LCP/physionet-build/pull/1602 and https://github.com/MIT-LCP/physionet-build/pull/1817