osmaa
commented
1 year ago 596 and ActivityPubFormatError apparently are not enough to fully solve pinned posts causing integrity violations. My latest stack trace on this issue below. I have not yet fully parsed this, posting for reference.
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 349, in main_wrap
raise exc_info[1]
File "/takahe/activities/models/post.py", line 996, in by_object_uri
return cls.objects.get(object_uri=object_uri)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 637, in get
raise self.model.DoesNotExist(
activities.models.post.Post.DoesNotExist: Post matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/takahe/stator/models.py", line 201, in transition_attempt
next_state = async_to_sync(current_state.handler)(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 277, in __call__
return call_result.result()
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 353, in main_wrap
result = await self.awaitable(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/takahe/users/models/identity.py", line 118, in handle_outdated
if await identity.fetch_actor():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/takahe/users/models/identity.py", line 906, in fetch_actor
await sync_to_async(service.sync_pins)(featured)
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 479, in __call__
ret: _R = await loop.run_in_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/asgiref/current_thread_executor.py", line 40, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 538, in thread_handler
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/takahe/users/services/identity.py", line 195, in sync_pins
post = Post.by_object_uri(object_uri, fetch=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/takahe/activities/models/post.py", line 1015, in by_object_uri
post = cls.by_ap(
^^^^^^^^^^
File "/takahe/activities/models/post.py", line 889, in by_ap
post.content = get_value_or_map(data, "content", "contentMap")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/takahe/core/ld.py", line 700, in get_value_or_map
raise ActivityPubFormatError(f"Cannot find {key} or {map_key}")
core.exceptions.ActivityPubFormatError: Cannot find content or contentMap
users.identity: 197352796249312186: outdated unchanged (0.90s)
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 349, in main_wrap
raise exc_info[1]
File "/takahe/activities/models/post.py", line 996, in by_object_uri
return cls.objects.get(object_uri=object_uri)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 637, in get
raise self.model.DoesNotExist(
activities.models.post.Post.DoesNotExist: Post matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/takahe/activities/models/post.py", line 836, in by_ap
post: Post = cls.objects.select_related("author__domain").get(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 637, in get
raise self.model.DoesNotExist(
activities.models.post.Post.DoesNotExist: Post matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/takahe/activities/models/post.py", line 859, in by_ap
post = cls.objects.select_related("author__domain").get(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 637, in get
raise self.model.DoesNotExist(
activities.models.post.Post.DoesNotExist: Post matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute
raise ex.with_traceback(None)
psycopg.errors.UniqueViolation: duplicate key value violates unique constraint "activities_post_object_uri_key"
DETAIL: Key (object_uri)=(https://todon.nl/users/vreer/statuses/109539916790724539) already exists.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/takahe/activities/models/post.py", line 865, in by_ap
post = cls.objects.create(
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 658, in create
obj.save(force_insert=True, using=self.db)
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
self.save_base(
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 877, in save_base
updated = self._save_table(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1020, in _save_table
results = self._do_insert(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1061, in _do_insert
return manager._insert(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1805, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/__init__.py", line 563, in execute
return real_execute(self, sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/psycopg/cursor.py", line 723, in execute
raise ex.with_traceback(None)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "activities_post_object_uri_key"
DETAIL: Key (object_uri)=(https://todon.nl/users/vreer/statuses/109539916790724539) already exists.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/takahe/stator/models.py", line 201, in transition_attempt
next_state = async_to_sync(current_state.handler)(self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 277, in __call__
return call_result.result()
^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 353, in main_wrap
result = await self.awaitable(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/takahe/users/models/identity.py", line 118, in handle_outdated
if await identity.fetch_actor():
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/takahe/users/models/identity.py", line 906, in fetch_actor
await sync_to_async(service.sync_pins)(featured)
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 479, in __call__
ret: _R = await loop.run_in_executor(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/asgiref/current_thread_executor.py", line 40, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 538, in thread_handler
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/takahe/users/services/identity.py", line 209, in sync_pins
for removed in PostInteraction.objects.filter(
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 398, in __iter__
self._fetch_all()
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1881, in _fetch_all
self._result_cache = list(self._iterable_class(self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
results = compiler.execute_sql(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/__init__.py", line 563, in execute
return real_execute(self, sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 83, in _execute
self.db.validate_no_broken_transaction()
File "/usr/local/lib/python3.11/site-packages/django/db/backends/base/base.py", line 531, in validate_no_broken_transaction
raise TransactionManagementError(
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
My instance (which I just rebooted with a blank database due to a mishap, oops -- this may be related, since the database content doesn't match presumed federation state) where the stack traces first have a lot of DoesNotExist, but then during processing a psycopg.errors.UniqueViolation and/or django.db.utils.IntegrityError happens due to constraint violation on index "activities_post_object_uri_key".
If I'm reading this correctly, the root cause is that actitivites.models.Post.by_ap() fetches the identity, (including pinned posts etc?), and immediately afterwards in the same transaction a blank post with the same id is created again.
I have an ugly patch for this, but I'm not opening a PR with it as it's probably not the correct fix.