michiya / django-pyodbc-azure

Django backend for Microsoft SQL Server and Azure SQL Database using pyodbc
https://pypi.python.org/pypi/django-pyodbc-azure
BSD 3-Clause "New" or "Revised" License
321 stars 140 forks source link

How to set Unicode for fields? #65

Open unformatt opened 8 years ago

unformatt commented 8 years ago

I'm looking to store Unicode, not just latin-1 which seems to be Microsoft's favorite charset.

my config

DATABASES = {
    'default': {
       'ENGINE': "sql_server.pyodbc",
       'USER': None,
       'PASSWORD': None,
       'NAME': None,
       'AUTOCOMMIT': True, 
       'OPTIONS': {
          'dsn': None,
          'autocommit': True, 
          'extra_params': 'tds_version=7.3', 
          'driver_charset': 'utf-8',
          'collation': 'utf-8',
          'unicode_results': False 
        }
    }
}

My field

label = models.CharField(db_column='Label', max_length=250, blank=True, null=True)

I run migrations to create DB table and this SQL is executed:

CREATE TABLE XXX ([id] int IDENTITY (1, 1) NOT NULL PRIMARY KEY, [Label] nvarchar(250) NULL;

DB shows "Label" collation as SQL_Latin1_General_CP1_CI_AS

So if I insert Unicode into "Label" column, it appears to be blank. And for constrast, If I change driver_charset and collation to "iso-8859-1", then I get this when inserting unicode into "Label" column:

  File "site-packages/sql_server/pyodbc/base.py", line 513, in format_params
    fp.append(smart_str(p, self.driver_charset))
  File "site-packages/django/utils/encoding.py", line 109, in smart_bytes
    return force_bytes(s, encoding, strings_only, errors)
  File "site-packages/django/utils/encoding.py", line 146, in force_bytes
    return s.encode(encoding, errors)
UnicodeEncodeError: 'latin-1' codec can't encode character u'\u2019' in position 8: ordinal not in range(256)
FlipperPA commented 8 years ago

Are you on Linux? If so, what version of the FreeTDS driver are you using? Python 2 or 3? Thanks.

unformatt commented 8 years ago

Yes, I'm on Ubuntu 14. FreeTDS conf says $Id: freetds.conf,v 1.12 2007/12/25 06:02:36 jklowden Exp and I'm using python 2.7

FlipperPA commented 8 years ago

Sorry, I should have been more clear. When you run tsql -C what version of FreeTDS does it tell you you are on? I'm guessing it'll most likely be 0.91 and 0.95.

unformatt commented 8 years ago
The program 'tsql' is currently not installed. You can install it by typing:
sudo apt-get install freetds-bin
FlipperPA commented 8 years ago

Do you have access to install it? We need to know what FreeTDS version you're running in order to know which TDS version to use in your configuration.

FlipperPA commented 8 years ago

Also, which package are you using to Django's odbc connection? Can you provide a link? django-pyodbc-azure is the one that has worked best for me, but the configs are slightly different for each.