search

digitalfabrik / integreat-cms

Simplified content management back end for the Integreat App - a multilingual information platform for newcomers
https://digitalfabrik.github.io/integreat-cms/
Apache License 2.0
56 stars 33 forks source link

Hide links from old versions in link checker #2466

Closed MizukiTemma closed 7 months ago

MizukiTemma commented 11 months ago

Describe the Bug

Internal server error will be caused during link replacement in the broken link list.

Steps to Reproduce

  1. Go to https://admin.integreat-app.de/lk-emmendingen/analytics/linkcheck/invalid/?page=2
  2. Find the link with linktext "Hauptstraße 15, 79341 Kenzingen"
  3. Replace the URL with "https://www.google.com/maps/place/Hauptstra%C3%9Fe+15,+79341+Kenzingen/@48.1918235,7.7665451,17z/data=!3m1!4b1!4m6!3m5!1s0x47913cec05b5a17f:0xe65c8600197c8b17!8m2!3d48.19182!4d7.76912!16s%2Fg%2F11c5fw72pv?entry=ttu"
  4. . See error

Expected Behavior

The link will be successfully updated.

Actual Behavior

Internal server error occurs

Additional Information

This issue is related to #2465

Traceback ``` Oct 16 09:24:55 ERROR django.request - 500 Internal Server Error: /lk-emmendingen/analytics/linkcheck/invalid/746380/ Traceback (most recent call last): File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "pagetranslation_unique_version" DETAIL: Key (page_id, language_id, version)=(14470, 74, 5) already exists. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner response = get_response(request) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/views/generic/base.py", line 70, in view return self.dispatch(request, *args, **kwargs) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/utils/decorators.py", line 43, in _wrapper return bound_method(*args, **kwargs) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view return view_func(request, *args, **kwargs) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/integreat_cms/cms/views/linkcheck/linkcheck_list_view.py", line 117, in dispatch return super().dispatch(request, *args, **kwargs) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch return handler(request, *args, **kwargs) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/integreat_cms/cms/views/linkcheck/linkcheck_list_view.py", line 189, in post new_translation.save() File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/integreat_cms/cms/models/abstract_content_translation.py", line 591, in save super().save(*args, **kwargs) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 739, in save self.save_base(using=using, force_insert=force_insert, File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base updated = self._save_table( File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert return manager._insert( File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert return query.get_compiler(using=using).execute_sql(returning_fields) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql cursor.execute(sql, params) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/cacheops/transaction.py", line 98, in execute result = self._no_monkey.execute(self, sql, params) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute return self._execute_with_wrappers(sql, params, many=False, executor=self._execute) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers return executor(sql, params, many, context) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__ raise dj_exc_value.with_traceback(traceback) from exc_value File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute return self.cursor.execute(sql, params) django.db.utils.IntegrityError: duplicate key value violates unique constraint "pagetranslation_unique_version" DETAIL: Key (page_id, language_id, version)=(14470, 74, 5) already exists. ```
timobrembeck commented 11 months ago

Hmm, the real source of the problem seems to be earlier (in the background job, not during link replacement):

Oct 16 09:06:57 ERROR   linkcheck.listeners - IntegrityError while running do_check_instance_links with args=(<class 'integreat_cms.cms.models.pages.page_translation.PageTranslation'>, <PageTranslation
(id: 5383498, page_id: 17315, language: de, slug: beratung)>, True) and kwargs={}: duplicate key value violates unique constraint "linkcheck_url_url_key"
DETAIL:  Key (url)=(https://www.hagen.de/irj/portal/FB-BRAR) already exists.
Traceback (most recent call last):
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "linkcheck_url_url_key"
DETAIL:  Key (url)=(https://www.hagen.de/irj/portal/FB-BRAR) already exists.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/linkcheck/listeners.py", line 33, in linkcheck_worker
    task['target'](*task['args'], **task['kwargs'])
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/linkcheck/listeners.py", line 107, in do_check_instance_links
    u.check_url()
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/linkcheck/models.py", line 282, in check_url
    return self.check_external(external_recheck_interval)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/linkcheck/models.py", line 469, in check_external
    self.save()
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/cacheops/transaction.py", line 98, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/integreat-cms/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "linkcheck_url_url_key"
DETAIL:  Key (url)=(https://www.hagen.de/irj/portal/FB-BRAR) already exists.

So I'll blame our inconsistent database at the moment, I don't think there is a new bug in the link checker, just the existing one (with the root cause https://github.com/DjangoAdminHackers/django-linkcheck/issues/168)

timobrembeck commented 10 months ago

Since this happened again after a few days, it seems to be relevant enough that we need a more informative error message here. I think we should point out that the link is from an old version and that the users need to wait a week (to make sure the linkcheck scan can fix the inconsistency) or contact their administrator.

PeterNerlich commented 7 months ago

Do you have a method to reproduce the bug locally?

timobrembeck commented 7 months ago

Do you have a method to reproduce the bug locally?

No, it's not possible to reproduce the race condition deterministically. However, you can just create a link object for an old translation version manually via the Django shell or via Django admin.