Bug: Inserting some characters into the database doesn't work

[cecilialau6776]

What happened?

I was trying to create a new project with this character: Δ, and was sent to the error page that says "We're having a bit of system trouble at the moment. Please check back soon!"

I expected the project creation to work, or give the user a response saying that a character isn't allowed.

Version

1.1.5

Component

Projects

What browsers are you seeing the problem on?

Other

Relevant log output

Internal Server Error: /project/create/
Traceback (most recent call last):
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/MySQLdb/cursors.py", line 331, in _query
    self._do_get_result(db)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/MySQLdb/cursors.py", line 136, in _do_get_result
    self._result = result = self._get_result()
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/MySQLdb/cursors.py", line 363, in _get_result
    return self._get_db().store_result()
MySQLdb.OperationalError: (1366, "Incorrect string value: '\\xCE\\x94-RNN...' for column `coldfront`.`project_project`.`description` at row 1")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/contrib/auth/mixins.py", line 71, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/contrib/auth/mixins.py", line 128, in dispatch
    return super().dispatch(request, *args, **kwargs)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/views/generic/edit.py", line 172, in post
    return super().post(request, *args, **kwargs)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/views/generic/edit.py", line 142, in post
    return self.form_valid(form)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/coldfront/core/project/views.py", line 468, in form_valid
    project_obj.save()
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/model_utils/models.py", line 38, in save
    super().save(*args, **kwargs)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/models/base.py", line 739, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/MySQLdb/cursors.py", line 331, in _query
    self._do_get_result(db)
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/MySQLdb/cursors.py", line 136, in _do_get_result
    self._result = result = self._get_result()
  File "/srv/coldfront/venv/lib64/python3.9/site-packages/MySQLdb/cursors.py", line 363, in _get_result
    return self._get_db().store_result()
django.db.utils.OperationalError: (1366, "Incorrect string value: '\\xCE\\x94-RNN...' for column `coldfront`.`project_project`.`description` at row 1")

Tasks/ user tests when bug is fixed

• [ ]

[meyersbs]

This just happened to me modifying a project description from the admin panel. It was the μ character.

[meyersbs]

And again with the ˚ character.

[meyersbs]

And again with ≅

[mdzik]

FYI: This seems to be rather DB (and MySQL in particular) related, not really Coldfront specific.

https://stackoverflow.com/questions/2108824/mysql-incorrect-string-value-error-when-save-unicode-string-in-django

[aebruno]

@meyersbs Are you also using MySQL/MariaDB? May be related to the default charset being set to latin1. See here.

You can check the default charset by running something like:

show create table project_project

If it's latin1 then you'll need to change it to something like utf8_general_ci. When creating the coldfront database in mysql you can also set this so all tables have it by default.