search

WeblateOrg / weblate

Web based localization tool with tight version control integration.
https://weblate.org/
GNU General Public License v3.0
4.54k stars 1.01k forks source link

SQL Incorrect string value with special characters #1054

Closed pc-jedi closed 8 years ago

pc-jedi commented 8 years ago

Steps to reproduce

  1. Add new component where source files container character like "Aktivních klanů"

    Actual behaviour

Environment:

Request Method: POST
Request URL: http://192.168.0.5:8000/admin/trans/subproject/add/

Django Version: 1.9.5
Python Version: 3.5.1
Installed Applications:
('django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.admin',
 'django.contrib.admindocs',
 'django.contrib.sitemaps',
 'social.apps.django_app.default',
 'crispy_forms',
 'compressor',
 'rest_framework',
 'rest_framework.authtoken',
 'weblate.trans',
 'weblate.lang',
 'weblate.accounts',
 'weblate')
Installed Middleware:
('django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.locale.LocaleMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'social.apps.django_app.middleware.SocialAuthExceptionMiddleware',
 'weblate.accounts.middleware.RequireLoginMiddleware')

Traceback:

File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py" in execute
  64.                 return self.cursor.execute(sql, params)

File "/usr/local/lib/python3.5/site-packages/django/db/backends/mysql/base.py" in execute
  112.             return self.cursor.execute(query, args)

File "/usr/local/lib/python3.5/site-packages/MySQLdb/cursors.py" in execute
  226.             self.errorhandler(self, exc, value)

File "/usr/local/lib/python3.5/site-packages/MySQLdb/connections.py" in defaulterrorhandler
  36.         raise errorvalue

File "/usr/local/lib/python3.5/site-packages/MySQLdb/cursors.py" in execute
  217.             res = self._query(query)

File "/usr/local/lib/python3.5/site-packages/MySQLdb/cursors.py" in _query
  378.         rowcount = self._do_query(q)

File "/usr/local/lib/python3.5/site-packages/MySQLdb/cursors.py" in _do_query
  341.         db.query(q)

File "/usr/local/lib/python3.5/site-packages/MySQLdb/connections.py" in query
  280.             _mysql.connection.query(self, query)

The above exception ((1366, "Incorrect string value: '\\xC5\\xAF' for column 'target' at row 1")) was the direct cause of the following exception:

File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py" in get_response
  149.                     response = self.process_exception_by_middleware(e, request)

File "/usr/local/lib/python3.5/site-packages/django/core/handlers/base.py" in get_response
  147.                     response = wrapped_callback(request, *callback_args, **callback_kwargs)

File "/usr/local/lib/python3.5/site-packages/django/contrib/admin/options.py" in wrapper
  541.                 return self.admin_site.admin_view(view)(*args, **kwargs)

File "/usr/local/lib/python3.5/site-packages/django/utils/decorators.py" in _wrapped_view
  149.                     response = view_func(request, *args, **kwargs)

File "/usr/local/lib/python3.5/site-packages/django/views/decorators/cache.py" in _wrapped_view_func
  57.         response = view_func(request, *args, **kwargs)

File "/usr/local/lib/python3.5/site-packages/django/contrib/admin/sites.py" in inner
  244.             return view(request, *args, **kwargs)

File "/usr/local/lib/python3.5/site-packages/django/contrib/admin/options.py" in add_view
  1437.         return self.changeform_view(request, None, form_url, extra_context)

File "/usr/local/lib/python3.5/site-packages/django/utils/decorators.py" in _wrapper
  67.             return bound_func(*args, **kwargs)

File "/usr/local/lib/python3.5/site-packages/django/utils/decorators.py" in _wrapped_view
  149.                     response = view_func(request, *args, **kwargs)

File "/usr/local/lib/python3.5/site-packages/django/utils/decorators.py" in bound_func
  63.                 return func.__get__(self, type(self))(*args2, **kwargs2)

File "/usr/local/lib/python3.5/contextlib.py" in inner
  30.                 return func(*args, **kwds)

File "/usr/local/lib/python3.5/site-packages/django/contrib/admin/options.py" in changeform_view
  1378.                 self.save_model(request, new_object, form, not add)

File "/usr/local/lib/python3.5/site-packages/django/contrib/admin/options.py" in save_model
  991.         obj.save()

File "/usr/local/lib/python3.5/site-packages/weblate/trans/models/subproject.py" in save
  1353.             self.create_translations()

File "/usr/local/lib/python3.5/site-packages/weblate/trans/models/subproject.py" in create_translations
  1000.                     self, lang, code, path, force, request=request

File "/usr/local/lib/python3.5/site-packages/weblate/trans/models/translation.py" in check_sync
  70.         translation.check_sync(force, request=request)

File "/usr/local/lib/python3.5/site-packages/weblate/trans/models/translation.py" in check_sync
  461.                 self, unit, pos

File "/usr/local/lib/python3.5/site-packages/weblate/trans/models/unit.py" in update_from_unit
  107.         dbunit.update_from_unit(unit, pos, created)

File "/usr/local/lib/python3.5/site-packages/weblate/trans/models/unit.py" in update_from_unit
  485.             same_state=same_state

File "/usr/local/lib/python3.5/site-packages/weblate/trans/models/unit.py" in save
  747.         super(Unit, self).save(*args, **kwargs)

File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py" in save
  708.                        force_update=force_update, update_fields=update_fields)

File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py" in save_base
  736.             updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)

File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py" in _save_table
  820.             result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)

File "/usr/local/lib/python3.5/site-packages/django/db/models/base.py" in _do_insert
  859.                                using=using, raw=raw)

File "/usr/local/lib/python3.5/site-packages/django/db/models/manager.py" in manager_method
  122.                 return getattr(self.get_queryset(), name)(*args, **kwargs)

File "/usr/local/lib/python3.5/site-packages/django/db/models/query.py" in _insert
  1039.         return query.get_compiler(using=using).execute_sql(return_id)

File "/usr/local/lib/python3.5/site-packages/django/db/models/sql/compiler.py" in execute_sql
  1060.                 cursor.execute(sql, params)

File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py" in execute
  79.             return super(CursorDebugWrapper, self).execute(sql, params)

File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py" in execute
  64.                 return self.cursor.execute(sql, params)

File "/usr/local/lib/python3.5/site-packages/django/db/utils.py" in __exit__
  95.                 six.reraise(dj_exc_type, dj_exc_value, traceback)

File "/usr/local/lib/python3.5/site-packages/django/utils/six.py" in reraise
  685.             raise value.with_traceback(tb)

File "/usr/local/lib/python3.5/site-packages/django/db/backends/utils.py" in execute
  64.                 return self.cursor.execute(sql, params)

File "/usr/local/lib/python3.5/site-packages/django/db/backends/mysql/base.py" in execute
  112.             return self.cursor.execute(query, args)

File "/usr/local/lib/python3.5/site-packages/MySQLdb/cursors.py" in execute
  226.             self.errorhandler(self, exc, value)

File "/usr/local/lib/python3.5/site-packages/MySQLdb/connections.py" in defaulterrorhandler
  36.         raise errorvalue

File "/usr/local/lib/python3.5/site-packages/MySQLdb/cursors.py" in execute
  217.             res = self._query(query)

File "/usr/local/lib/python3.5/site-packages/MySQLdb/cursors.py" in _query
  378.         rowcount = self._do_query(q)

File "/usr/local/lib/python3.5/site-packages/MySQLdb/cursors.py" in _do_query
  341.         db.query(q)

File "/usr/local/lib/python3.5/site-packages/MySQLdb/connections.py" in query
  280.             _mysql.connection.query(self, query)

Exception Type: OperationalError at /admin/trans/subproject/add/
Exception Value: (1366, "Incorrect string value: '\\xC5\\xAF' for column 'target' at row 1")

Local variables:

sql 
('INSERT INTO `trans_unit` (`translation_id`, `checksum`, `contentsum`, '
 '`location`, `context`, `comment`, `flags`, `source`, `previous_source`, '
 '`target`, `fuzzy`, `translated`, `position`, `has_suggestion`, '
 '`has_comment`, `has_failing_check`, `num_words`, `priority`) VALUES (%s, %s, '
 '%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)')
params  
[2,
 '51f1d88b10f0a4f36e88df0a299d24de',
 '51f1d88b10f0a4f36e88df0a299d24de',
 "$lang->'activeclans'",
 '',
 '',
 '',
 'Active clans',
 '',
 'Aktivních klanů',
 False,
 True,
 2,
 False,
 False,
 False,
 2,
 100]

Server configuration

 * Weblate 2.6-dev
 * Python 3.5.1
 * Django 1.9.5
 * six 1.10.0
 * python-social-auth 0.2.14
 * Translate Toolkit 1.14.0-rc1
 * Whoosh 2.7.4
 * Git 2.1.4
 * Pillow (PIL) 1.1.7
 * dateutil 2.5.2
 * lxml 3.6.0
 * django-crispy-forms 1.6.0
 * compressor 1.6
 * djangorestframework 3.3.3
 * pyuca N/A
 * pyLibravatar N/A
 * Database backends: django.db.backends.mysql
nijel commented 8 years ago

Is your database using UTF-8? AFAIK it's default with recent MariaDB servers, but for MySQL it has to be turned on, see https://docs.weblate.org/en/latest/admin/install.html#creating-database-in-mysql

pc-jedi commented 8 years ago

Your are right, I overlooked the obvious. I had a typo in "utf8_general_ci"