search

linuxlewis / djorm-ext-pgfulltext

PostgreSQL full-text search integration with django orm.
Other
250 stars 84 forks source link

0.9.3 UnicodeEncodeError #52

Open aisayko opened 9 years ago

aisayko commented 9 years ago

Hello,

I just upgraded djorm-ext-pgfulltext to version 0.9.3 and got errors when use russian chars on my search query.

In version 0.9.2 it was okay.

Raising error:

Traceback (most recent call last):
  File "/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "./apps/utils/views.py", line 98, in autocomplete
    'type': 'Hotels'} for o in hqs])
  File "//local/lib/python2.7/site-packages/django/db/models/query.py", line 141, in __iter__
    self._fetch_all()
  File "/home/bla/local/lib/python2.7/site-packages/django/db/models/query.py", line 966, in _fetch_all
    self._result_cache = list(self.iterator())
  File "/home/bla/local/lib/python2.7/site-packages/django/db/models/query.py", line 265, in iterator
    for row in compiler.results_iter():
  File "/home/bla/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 700, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/home/bla/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 775, in execute_sql
    sql, params = self.as_sql()
  File "/home/bla/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 109, in as_sql
    where, w_params = self.compile(self.query.where)
  File "/home/bla/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 80, in compile
    return node.as_sql(self, self.connection)
  File "/home/bla/local/lib/python2.7/site-packages/django/db/models/sql/where.py", line 106, in as_sql
    sql, params = qn.compile(child)
  File "/home/bla/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 80, in compile
    return node.as_sql(self, self.connection)
  File "/home/bla/local/lib/python2.7/site-packages/djorm_pgfulltext/fields.py", line 87, in as_sql
    rest = (" & ".join(self.transform.__call__(rhs_params)),)
  File "/home/bla/local/lib/python2.7/site-packages/djorm_pgfulltext/fields.py", line 139, in transform
    return startswith(*args)
  File "/home/bla/local/lib/python2.7/site-packages/djorm_pgfulltext/fields.py", line 61, in startswith
    return [x + ":*" for x in quotes(wordlist)]
  File "/home/bla/local/lib/python2.7/site-packages/djorm_pgfulltext/fields.py", line 58, in quotes
    return ["%s" % adapt(x.replace("\\", "")) for x in wordlist]
UnicodeEncodeError: 'latin-1' codec can't encode characters in position 0-7: ordinal not in range(256)
stefanw commented 9 years ago

Got something similar with a German ß. Encoding the query as utf-8 before handing it to the full text query fixed this. However, this is not how it should be. Encoding should happen as late as possible and certainly not in a Django ORM query.

psycopg2.extensions.adapt apparently needs an utf-8 encoded string here.