Error with results containing non-ASCII characters

[GoogleCodeExporter]

I was running a query on a MSSQL server where one of the column has non-ASCII 
characters on the result. The python traceback on the console is:

Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/cf/ui/editor.py", line 214, in on_query_finished
    self.results.set_query(query)
  File "/usr/lib/pymodules/python2.6/cf/ui/editor.py", line 943, in set_query
    self.grid.set_query(query)
  File "/usr/lib/pymodules/python2.6/cf/ui/editor.py", line 1053, in set_query
    self.query.coding_hint)
  File "/usr/lib/pymodules/python2.6/cf/ui/widgets/grid.py", line 499, in set_result
    self._setup_columns(rows)
  File "/usr/lib/pymodules/python2.6/cf/ui/widgets/grid.py", line 118, in _setup_columns
    col.set_fixed_width(self._get_best_column_width(i, samples))
  File "/usr/lib/pymodules/python2.6/cf/ui/widgets/grid.py", line 139, in _get_best_column_width
    value = model._get_markup_for_value(row[colnum], markup=False)
  File "/usr/lib/pymodules/python2.6/cf/ui/widgets/grid.py", line 577, in _get_markup_for_value
    value = unicode(value, self.coding_hint)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xed in position 16: invalid 
continuation byte

The result is that the column which contains the non-ASCII character, and the 
rest of the columns beyond that one, doesn't appear in the result windows. Even 
the result windows is not shown automatically.

If in the query I delete the non-ASCII column everything works as expected.

Please let me know if you need any more information from me.

Thanks and excuse for my english but it's not my native language.
Ernesto.

Original issue reported on code.google.com by edo...@gmail.com on 10 Nov 2010 at 8:28

[GoogleCodeExporter]

Could you try the following:

Open ~/.config/crunchyfrog/datasources.cfg with a text editor.
Look for an URL in this file that describes your connection (a line
starting with "url = mssql://").
Append "?charset=utf-8" or "?charset=utf8" at the end of this URL.
Restart CrunchyFrog and try your query again.

I don't have access to a SQL Server instance, so I can't test it myself
ATM.

Thanks!

Original comment by albrecht.andi on 11 Nov 2010 at 11:35

• Added labels: Backend-SQLServer, FeedbackRequested

[GoogleCodeExporter]

I tried with those two options but the result was worst. It don't even show any 
column and the Traceback was:

Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.6/cf/ui/editor.py", line 344, in exec_threaded
    query.execute(True)
  File "/usr/lib/pymodules/python2.6/cf/db/__init__.py", line 660, in execute
    self.rows = dbapi_cur.fetchall()
  File "/usr/lib/pymodules/python2.6/pymssql.py", line 307, in fetchall
    list = [tuple([row[r] for r in sorted(row.keys()) if type(r) == int]) for row in self._source]
  File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe9 in position 5: invalid 
continuation byte

Anyway, I did some queries on a console with an script and found that the 
column that produce this behavior is on Latin1 so I changed the datasouce to 
charset=latin1 and it worked perfectly.

So, ¿there's some way to autodetect the encoding of the schema to use the 
appropriate encoding?, or at least one should have some option to chose it.

Thanks.
Ernesto

Original comment by edo...@gmail.com on 11 Nov 2010 at 1:57

[GoogleCodeExporter]

Adding an option when defining the datasource sounds reasonable. Thanks for the 
feedback!

Original comment by albrecht.andi on 11 Nov 2010 at 2:19

• Changed state: Accepted
• Added labels: Milestone-0.4.2
• Removed labels: FeedbackRequested

[GoogleCodeExporter]

This issue was closed by revision bf560fad0b.

Original comment by albrecht.andi on 14 Nov 2010 at 8:02

• Changed state: Fixed