import_project slug URL computation creates duplicate keys and fails

[stormi]

When creating subprojects with import_project, if the subprojects have a long name, it can happen that the URL slug computed by weblate from the name is not unique and already exists in the database. This causes the operation to fail.

I got this issue because I'm trying to use weblate with web projects that have lots of i18n files all over the place and I chose to put the path in the subproject name.

Traceback (most recent call last):
  File "manage.py", line 31, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/opt/bitnami/apps/weblate/weblate/trans/management/commands/import_project.py", line 229, in handle
    filemask=self.filemask.replace('**', match)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/db/models/manager.py", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/db/models/query.py", line 372, in create
    obj.save(force_insert=True, using=self.db)
  File "/opt/bitnami/apps/weblate/weblate/trans/models/subproject.py", line 1267, in save
    super(SubProject, self).save(*args, **kwargs)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/db/models/base.py", line 589, in save
    force_update=force_update, update_fields=update_fields)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/db/models/base.py", line 617, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/db/models/base.py", line 698, in _save_table
    result = self._do_insert(cls._base_manager, using, fields, update_pk, raw)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/db/models/base.py", line 731, in _do_insert
    using=using, raw=raw)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/db/models/manager.py", line 92, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/db/models/query.py", line 921, in _insert
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/db/models/sql/compiler.py", line 921, in execute_sql
    cursor.execute(sql, params)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/opt/bitnami/apps/django/lib/python2.7/site-packages/Django-1.7.8-py2.7.egg/django/db/backends/mysql/base.py", line 129, in execute
    return self.cursor.execute(query, args)
  File "build/bdist.linux-x86_64/egg/MySQLdb/cursors.py", line 173, in execute
  File "build/bdist.linux-x86_64/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
django.db.utils.IntegrityError: (1062, "Duplicate entry '2-oostats-frontend-src_app_widgets_users_connections' for key 'trans_subproject_project_id_427a70a765c3a95c_uniq'")

Server configuration

# ./manage.py list_versions
/opt/bitnami/python/lib/python2.7/site-packages/setuptools-7.0-py2.7.egg/pkg_resources.py:1045: UserWarning: /opt/bitnami/.tmp is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).
 * Weblate 2.3
 * Python 2.7.9
 * Django 1.7.8
 * python-social-auth 0.2.1
 * Translate Toolkit 1.13.0
 * Whoosh 2.6.0
 * Git 1.9.5
 * Pillow (PIL) 1.1.7
 * dateutil 1.5
 * lxml 3.3.6
 * django-crispy-forms 1.4.0

[nijel]

What is expected behavior in this case?

[nijel]

Also this happens only with MySQL which does silent truncation in this case, I think other databases would complain when creating entry with field which doesn't fit into the database...

[nijel]

Weblate is supposed to skip already existing slugs. The reason for this is to allow running import_project several times without creating duplicate components.

[stormi]

It's too bad that the slug is a unique key but is at the same time limited in size. If weblate ignores already existing slugs, it will ignore some components although they never got imported, because their truncated slug is identical.

[nijel]

As it's used in the URL, it's size has be limited. If not something else, there is limit on URL length and we need to pass other things in URL (eg. search parameters).

Also as the slug is calculated from name on import and the too long names won't fit in the UI anyway, so the best approach IMHO is to have shorter names...

[stormi]

Ok, thanks for the answer.

[nijel]

Anyway I think this could be configurable in the end by command line parameter.