Tour table search: "Illegal mix of collations"

[hyanwong]

There's an error like this in the logs. I assume we have the wrong collation setting on one of the columns in the tour table on production.

Traceback (most recent call last):
  File "/usr/home/web2py/www.onezoom.org/gluon/restricted.py", line 219, in restricted
    exec(ccode, environment)
  File "/usr/home/web2py/www.onezoom.org/applications/OZtree/compiled/controllers.tour.search.py", line 339, in <module>
  File "/usr/home/web2py/www.onezoom.org/gluon/globals.py", line 463, in <lambda>
    self._caller = lambda f: f()
  File "/usr/home/web2py/www.onezoom.org/applications/OZtree/compiled/controllers.tour.search.py", line 329, in search
  File "/usr/home/web2py/www.onezoom.org/applications/OZtree/modules/tour.py", line 47, in tour_search
    return db(query).select(db.tour.ALL, orderby=(db.tour.title))
  File "/usr/home/web2py/www.onezoom.org/gluon/packages/dal/pydal/objects.py", line 2686, in select
    return adapter.select(self.query, fields, attributes)
  File "/usr/home/web2py/www.onezoom.org/gluon/packages/dal/pydal/adapters/base.py", line 885, in select
    return self._select_aux(sql, fields, attributes, colnames)
  File "/usr/home/web2py/www.onezoom.org/gluon/packages/dal/pydal/adapters/base.py", line 842, in _select_aux
    rows = self._select_aux_execute(sql)
  File "/usr/home/web2py/www.onezoom.org/gluon/packages/dal/pydal/adapters/base.py", line 836, in _select_aux_execute
    self.execute(sql)
  File "/usr/home/web2py/www.onezoom.org/gluon/packages/dal/pydal/adapters/__init__.py", line 70, in wrap
    return f(*args, **kwargs)
  File "/usr/home/web2py/www.onezoom.org/gluon/packages/dal/pydal/adapters/base.py", line 446, in execute
    rv = self.cursor.execute(command, *args[1:], **kwargs)
  File "/usr/home/web2py/www.onezoom.org/lib/python3.10/site-packages/pymysql/cursors.py", line 153, in execute
    result = self._query(query)
  File "/usr/home/web2py/www.onezoom.org/lib/python3.10/site-packages/pymysql/cursors.py", line 322, in _query
    conn.query(q)
  File "/usr/home/web2py/www.onezoom.org/lib/python3.10/site-packages/pymysql/connections.py", line 558, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/usr/home/web2py/www.onezoom.org/lib/python3.10/site-packages/pymysql/connections.py", line 822, in _read_query_result
    result.read()
  File "/usr/home/web2py/www.onezoom.org/lib/python3.10/site-packages/pymysql/connections.py", line 1200, in read
    first_packet = self.connection._read_packet()
  File "/usr/home/web2py/www.onezoom.org/lib/python3.10/site-packages/pymysql/connections.py", line 772, in _read_packet
    packet.raise_for_error()
  File "/usr/home/web2py/www.onezoom.org/lib/python3.10/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/usr/home/web2py/www.onezoom.org/lib/python3.10/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.OperationalError: (1267, "Illegal mix of collations (utf8_general_ci,IMPLICIT) and (utf8mb4_general_ci,COERCIBLE) for operation 'like'")

Looks like it is triggered by visiting /tour/search.json?query=%F0%9F%90%85 (referred from https://www.onezoom.org/life/@Mammalia=244265?otthome=%40%3D770311)

[hyanwong]

Looks like it's triggered by searching for 4-byte emojis. I think in this case the query string was:

searchFor='🐅', language='en-IN,en-GB;q=0.9,en-US;q=0.8,en;q=0.7,te;q=0.6'

And indeed, 🐅 is encoded as %F0%9F%90%85

And you can trigger the error by going to

https://beta.onezoom.org/tour/search.json?query=🐅

[hyanwong]

I think all searchable columns should probably be utf8mb4 (not utf8mb3). We should probably add stuff to https://github.com/OneZoom/OZtree/blob/main/OZprivate/ServerScripts/SQL/create_db_indexes.sql to set the default encoding of character data in the tour table to utf8mb4, and also perform the check in tests/site_setup/test_database_settings.py?