"value too long for type character varying" when adding a variant

burner1024 commented 1 year ago

Describe the issue

An exception when adding a variant to a specific string.

I already tried

[X] I've read and searched the documentation.
[X] I've searched for similar issues in this repository.

Steps to reproduce the behavior

Try to add a variant. A long one, I guess.

Expected behavior

No exception, variant added.

Screenshots

No response

Exception traceback

Internal Server Error: /new-unit/infinity-engine/ascension/pl/

DataError at /new-unit/infinity-engine/ascension/pl/
value too long for type character varying(576)

Request Method: POST
Request URL: https://hive.bgforge.net/new-unit/infinity-engine/ascension/pl/
Django Version: 4.0.4
Python Executable: /usr/local/bin/python
Python Version: 3.10.4
Python Path: ['/', '/usr/local/lib/python3.10/site-packages/git/ext/gitdb', '/', '/usr/local/bin', '/usr/local/lib/python310.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-dynload', '/usr/local/lib/python3.10/site-packages', '/app/data/python', '/usr/local/lib/python3.10/site-packages/gitdb/ext/smmap']
Server time: Thu, 25 May 2023 18:12:42 +0000
Installed Applications:
['customize',
 'weblate.addons',
 'weblate.auth',
 'weblate.checks',
 'weblate.formats',
 'weblate.glossary',
 'weblate.machinery',
 'weblate.trans',
 'weblate.lang',
 'weblate_language_data',
 'weblate.memory',
 'weblate.screenshots',
 'weblate.fonts',
 'weblate.accounts',
 'weblate.configuration',
 'weblate.utils',
 'weblate.vcs',
 'weblate.wladmin',
 'weblate.metrics',
 'weblate',
 'weblate.gitexport',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.admin.apps.SimpleAdminConfig',
 'django.contrib.admindocs',
 'django.contrib.sitemaps',
 'django.contrib.humanize',
 'social_django',
 'crispy_forms',
 'compressor',
 'rest_framework',
 'rest_framework.authtoken',
 'django_filters']
Installed Middleware:
['weblate.middleware.RedirectMiddleware',
 'weblate.middleware.ProxyMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'weblate.accounts.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'social_django.middleware.SocialAuthExceptionMiddleware',
 'weblate.accounts.middleware.RequireLoginMiddleware',
 'weblate.api.middleware.ThrottlingMiddleware',
 'weblate.middleware.SecurityMiddleware']

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 657, in get_or_create
    return self.get(**kwargs), False
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 496, in get
    raise self.model.DoesNotExist(

During handling of the above exception (Variant matching query does not exist.), another exception occurred:
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)

The above exception (value too long for type character varying(576)
) was the direct cause of the following exception:
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.10/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/django/views.py", line 67, in sentry_wrapped_callback
    return callback(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/views/decorators/http.py", line 43, in inner
    return func(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/contrib/auth/decorators.py", line 23, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/weblate/trans/views/edit.py", line 947, in new_unit
    created_unit = translation.add_unit(request, **form.as_kwargs())
  File "/usr/local/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.10/site-packages/weblate/trans/models/translation.py", line 1376, in add_unit
    unit.save(
  File "/usr/local/lib/python3.10/site-packages/weblate/trans/models/unit.py", line 419, in save
    self.update_variants()
  File "/usr/local/lib/python3.10/site-packages/weblate/trans/models/unit.py", line 551, in update_variants
    variant = Variant.objects.get_or_create(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 664, in get_or_create
    return self.create(**params), True
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 514, in create
    obj.save(force_insert=True, using=self.db)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 806, in save
    self.save_base(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 857, in save_base
    updated = self._save_table(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 1000, in _save_table
    results = self._do_insert(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 1041, in _do_insert
    return manager._insert(
  File "/usr/local/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 1434, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1621, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.10/site-packages/sentry_sdk/integrations/django/__init__.py", line 544, in execute
    return real_execute(self, sql, params)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 67, in execute
    return self._execute_with_wrappers(
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
    with self.db.wrap_database_errors:
  File "/usr/local/lib/python3.10/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 89, in _execute
    return self.cursor.execute(sql, params)

Exception Type: DataError at /new-unit/infinity-engine/ascension/pl/
Exception Value: value too long for type character varying(576)

How do you run Weblate?

Docker container

Weblate versions

Weblate: 4.12.2
Django: 4.0.4
siphashc: 2.1
translate-toolkit: 3.6.2
lxml: 4.6.5
Pillow: 9.1.0
bleach: 5.0.0
python-dateutil: 2.8.2
social-auth-core: 4.2.0
social-auth-app-django: 5.0.0
django-crispy-forms: 1.14.0
oauthlib: 3.2.0
django-compressor: 4.0
djangorestframework: 3.13.1
django-filter: 21.1
django-appconf: 1.0.5
user-agents: 2.2.0
filelock: 3.6.0
jellyfish: 0.9.0
openpyxl: 3.0.9
celery: 5.2.6
kombu: 5.2.4
translation-finder: 2.12
weblate-language-data: 2022.2
html2text: 2020.1.16
pycairo: 1.21.0
pygobject: 3.42.1
diff-match-patch: 20200713
requests: 2.27.1
django-redis: 5.2.0
hiredis: 2.0.0
sentry_sdk: 1.5.12
Cython: 0.29.28
misaka: 2.1.1
GitPython: 3.1.27
borgbackup: 1.2.0
pyparsing: 3.0.9
pyahocorasick: 1.4.4
python-redis-lock: 3.7.0
charset-normalizer: 2.0.12
Python: 3.10.4
Git: 2.30.2
psycopg2: 2.9.3
psycopg2-binary: 2.9.3
phply: 1.2.5
ruamel.yaml: 0.17.21
tesserocr: 2.5.2
akismet: 1.1
boto3: 1.22.11
zeep: 4.1.0
aeidon: 1.11
iniparse: 0.5
mysqlclient: 2.1.0
Mercurial: 6.1.2
git-svn: 2.30.2
git-review: 2.3.1
Redis server: 6.2.7
PostgreSQL server: 14.2
Database backends: django.db.backends.postgresql
Cache backends: default:RedisCache, avatar:FileBasedCache
Email setup: django.core.mail.backends.smtp.EmailBackend: email-smtp.us-east-1.amazonaws.com
OS encoding: filesystem=utf-8, default=utf-8
Celery: redis://cache:6379/1, redis://cache:6379/1, regular
Platform: Linux 4.15.0-194-generic (x86_64)

Weblate deploy checks

System check identified some issues:

INFOS:
?: (weblate.I031) New Weblate version is available, please upgrade to 4.17.
    HINT: https://docs.weblate.org/en/weblate-4.12.2/admin/upgrade.html

System check identified 1 issue (2 silenced).

Additional context

No response

nijel commented 1 year ago

Variant length is currently limited to 576 chars:

https://github.com/WeblateOrg/weblate/blob/aa365cf9673a8b1c84773174637b1e1e113c42e2/weblate/trans/models/variant.py#L13-L15

PS: Similar limitation is there on PostgreSQL, see https://stackoverflow.com/q/41971217/225718

burner1024 commented 1 year ago

Well, 2712 is way more than 576. I don't see the point of setting the limitation explicitly in code, if the server throws an error anyway. To get around this, people seem to be using md5 or something.

Since I'm on Postgresql, may I raise this limit to 2712 safely? Or that can break future migrations?

nijel commented 1 year ago

I don't see the point of setting the limitation explicitly in code, if the server throws an error anyway.

Making it fail gracefully is definitely desired, it just does not happen here.

Since I'm on Postgresql, may I raise this limit to 2712 safely? Or that can break future migrations?

You can manually change the table schema, that should not break future migrations. The actual maximal value probably will be smaller as there are more fields in the index.

To get around this, people seem to be using md5 or something.

Yes, that's an option. Unfortunately, that is hard to make it compatible with all supported databases without introducing an additional field for storing that and updating it manually.

nijel commented 4 months ago

As Django now supports MD5 indexes, and we use them already, this might be the way to remove the limitation (and improve variants update performance, see https://github.com/WeblateOrg/weblate/issues/7661).

github-actions[bot] commented 4 months ago

This issue has been added to the backlog. It is not scheduled on the Weblate roadmap, but it eventually might be implemented.

In case you need this feature soon, please consider helping or push it by funding the development.

WeblateOrg / weblate