tethysplatform / tethys

The Tethys Platform main Django website project repository.
http://tethysplatform.org/
BSD 2-Clause "Simplified" License
94 stars 51 forks source link

[BUG] Some admin pages db operations are failing #1040

Closed sdc50 closed 7 months ago

sdc50 commented 7 months ago

Describe the bug I believe this is related to Django 4.

When trying to update a Group with app-level permissions in the admin pages:

```python Traceback (most recent call last): File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/asgiref/sync.py", line 472, in thread_handler raise exc_info[1] File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/core/handlers/exception.py", line 38, in inner response = await get_response(request) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/core/handlers/base.py", line 233, in _get_response_async response = await wrapped_callback(request, *callback_args, **callback_kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/asgiref/sync.py", line 435, in __call__ ret = await asyncio.wait_for(future, timeout=None) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/asyncio/tasks.py", line 408, in wait_for return await fut File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 22, in run result = self.fn(*self.args, **self.kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/asgiref/sync.py", line 476, in thread_handler return func(*args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/options.py", line 616, in wrapper return self.admin_site.admin_view(view)(*args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/utils/decorators.py", line 130, in _wrapped_view response = view_func(request, *args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func response = view_func(request, *args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/sites.py", line 232, in inner return view(request, *args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1660, in change_view return self.changeform_view(request, object_id, form_url, extra_context) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper return bound_method(*args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/utils/decorators.py", line 130, in _wrapped_view response = view_func(request, *args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1540, in changeform_view return self._changeform_view(request, object_id, form_url, extra_context) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1581, in _changeform_view new_object = self.save_form(request, form, change=not add) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1093, in save_form return form.save(commit=False) File "/Users/sdc50/Workspace/tethys/tethys-dev/tethys_apps/admin.py", line 412, in save for perm in remove_perms_query: File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/models/query.py", line 280, in __iter__ self._fetch_all() File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/models/query.py", line 1324, in _fetch_all self._result_cache = list(self._iterable_class(self)) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/models/query.py", line 51, in __iter__ results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1162, in execute_sql sql, params = self.as_sql() File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 522, in as_sql result, params = self.get_combinator_sql(combinator, self.query.combinator_all) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 459, in get_combinator_sql raise DatabaseError('ORDER BY not allowed in subqueries of compound statements.') django.db.utils.DatabaseError: ORDER BY not allowed in subqueries of compound statements. ```

I believe the culprit code is here, and specifically the difference method.

This alternative code seems to fix the error in my case, but I don't understand why it's not working in the first place:

remove_perms_query = set(self.fields[field].initial) - set(assign_perms_query)

Also, adding a ProxyApp fails with:

```python Traceback (most recent call last): File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/asgiref/sync.py", line 472, in thread_handler raise exc_info[1] File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/core/handlers/exception.py", line 38, in inner response = await get_response(request) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/core/handlers/base.py", line 233, in _get_response_async response = await wrapped_callback(request, *callback_args, **callback_kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/asgiref/sync.py", line 435, in __call__ ret = await asyncio.wait_for(future, timeout=None) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/asyncio/tasks.py", line 408, in wait_for return await fut File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/asgiref/current_thread_executor.py", line 22, in run result = self.fn(*self.args, **self.kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/asgiref/sync.py", line 476, in thread_handler return func(*args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/options.py", line 616, in wrapper return self.admin_site.admin_view(view)(*args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/utils/decorators.py", line 130, in _wrapped_view response = view_func(request, *args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/views/decorators/cache.py", line 44, in _wrapped_view_func response = view_func(request, *args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/sites.py", line 232, in inner return view(request, *args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1657, in add_view return self.changeform_view(request, None, form_url, extra_context) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/utils/decorators.py", line 43, in _wrapper return bound_method(*args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/utils/decorators.py", line 130, in _wrapped_view response = view_func(request, *args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1540, in changeform_view return self._changeform_view(request, object_id, form_url, extra_context) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1586, in _changeform_view self.save_model(request, new_object, form, not add) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/contrib/admin/options.py", line 1099, in save_model obj.save() File "/Users/sdc50/Workspace/tethys/tethys-dev/tethys_apps/models.py", line 1337, in save self.register_app_permission() File "/Users/sdc50/Workspace/tethys/tethys-dev/tethys_apps/models.py", line 1365, in register_app_permission p.save() File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/models/base.py", line 739, in save self.save_base(using=using, force_insert=force_insert, File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/models/base.py", line 776, in save_base updated = self._save_table( File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/models/base.py", line 881, in _save_table results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/models/base.py", line 919, in _do_insert return manager._insert( File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/models/query.py", line 1270, in _insert return query.get_compiler(using=using).execute_sql(returning_fields) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql cursor.execute(sql, params) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/backends/utils.py", line 98, in execute return super().execute(sql, params) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers return executor(sql, params, many, context) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute with self.db.wrap_database_errors: File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/Users/sdc50/conda/envs/tethys/lib/python3.10/site-packages/django/db/backends/sqlite3/base.py", line 423, in execute return Database.Cursor.execute(self, query, params) django.db.utils.IntegrityError: UNIQUE constraint failed: auth_permission.content_type_id, auth_permission.codename ```

I think this might only happen after the first one, because the root issue is that the permission_codename is ":access_app" due to the fact that self.package is the empty string. the self.__package property is supposed to be set to self.name on instantiation, but for some reason self.name is the empty string with __init__ is run. This only seems to be the case when created from the admin pages. When I directly instantiate a ProxyApp it works as expected. Again, I don't understand why the name field is not being set on instantiation from the admin pages, but a workaround is to modify the save method:

    def save(self, *args, **kwargs):
        if not self.package:
            self.__package = self.name
        ....

I'm not sure how self.name is being populated after __init__ but before save, but that seems to be what is happening.