UnicodeDecodeError: 'utf8' codec can't decode bytes, unexpected end of data, latin1

[GoogleCodeExporter]

What steps will reproduce the problem?
1.
Moldel

class PrSecurityBas(models.Model):
    username = models.TextField(db_column='USERNAME', primary_key=True ) # Field name made lowercase.
    infraest = models.TextField(db_column='INFRAEST' ) # Field name made lowercase.
    software = models.TextField(db_column='SOFTWARE' ) # Field name made lowercase.
    dni = models.CharField(max_length=15, db_column='DNI' ) # Field name made lowercase.
    apellidoynombres = models.TextField(db_column='APELLIDOYNOMBRES') # Field name made lowercase.
    class Meta:
        db_table = u'PR_SECURITY_BAS'

I run python manage.py shell

>>> from frontend.models import PrSecurityBas
>>> ll = PrSecurityBas.objects.all()
>>> for i in ll:
...     print i.username
...
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/usr/lib/pymodules/python2.6/django/db/models/query.py", line 106, in _result_iter
    self._fill_cache()
  File "/usr/lib/pymodules/python2.6/django/db/models/query.py", line 692, in _fill_cache
    self._result_cache.append(self._iter.next())
  File "/usr/lib/pymodules/python2.6/django/db/models/query.py", line 238, in iterator
    for row in self.query.results_iter():
  File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py", line 287, in results_iter
    for rows in self.execute_sql(MULTI):
  File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py", line 2389, in execute_sql
    return list(result)
  File "/usr/lib/pymodules/python2.6/django/db/models/sql/query.py", line 2383, in <lambda>
    result = iter((lambda: cursor.fetchmany(GET_ITERATOR_CHUNK_SIZE)),
  File "/usr/local/lib/python2.6/dist-packages/sql_server/pyodbc/base.py", line 350, in fetchmany
    return [self.format_results(row) for row in self.cursor.fetchmany(chunk)]
  File "/usr/local/lib/python2.6/dist-packages/sql_server/pyodbc/base.py", line 338, in format_results
    fr.append(row.decode('utf-8'))
  File "/usr/lib/python2.6/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 17-18: 
unexpected end of data

the db encoding is latin1
TABLE [dbo].[PR_SECURITY_BAS] (
    [USERNAME] [varchar] (20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
    [INFRAEST] [varchar] (20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
    [SOFTWARE] [varchar] (20) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
    [DNI] [char] (15) COLLATE SQL_Latin1_General_CP1_CI_AS NULL ,
    [APELLIDOYNOMBRES] [varchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NULL)

2.
BUT if I use :
>>> from django.db import connection
>>> cur = connection.cursor()
>>> r = cur.execute("select * from PR_SECURITY_BAS")
>>> print r.fetchall()

This return data OK with strings like this: 'Diez,Nolas Mart\xedn'

What is the expected output? What do you see instead?
I need to get data with objects.all() and show it.

What version of the product are you using? On what operating system?

-django-pyodbc-read-only (was installed two days ago)
-pyodbc-2.1.7
-freetds 0.82-6build1
-unixodbc  2.2.11-21
-MS Sql Server 2000

Please provide any additional information below.

sorry for my bad English

Original issue reported on code.google.com by derlokep...@gmail.com on 17 Apr 2010 at 4:17

[GoogleCodeExporter]

I Fix it changing te line 338 of the ???/sql_server/pyodbc/base.py file. 
replacing 
utf-8 for latin-1

Original comment by derlokep...@gmail.com on 22 Apr 2010 at 11:38

[GoogleCodeExporter]

Same thing worked for me, line 344. Could we make this a database option?

Thanks,
Collin

Original comment by CollinMA...@gmail.com on 24 Jun 2011 at 9:03

[GoogleCodeExporter]

I Fix using a workaround in my view.

...
conn = connections['MY_CONNECTION']
conn.driver_needs_utf8 = False
cursor = conn.cursor()
cursor.execute("SELECT * FROM some_table")
for row in cursor.fetchall():
    print(row)
...

Original comment by ramiro...@gmail.com on 15 Sep 2011 at 6:04

[GoogleCodeExporter]

See attached a patch to check 'uses_utf8' in your database's OPTIONS, if set to 
False pyodbc won't attempt to encode the results. Saves you from having to set 
driver_needs_utf8 on the fly.

YMMV but it works nicely for me.

Original comment by alexdela...@gmail.com on 9 Mar 2012 at 11:05

Attachments:

• django-pyodbc-uses_utf8.diff