search

ashishps1 / pyodbc

Automatically exported from code.google.com/p/pyodbc
MIT No Attribution
3 stars 0 forks source link

Character return ??? instead kanji (32 bit work - 3.0.3, 64 bit not - 3.0.6) #324

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago 
Hello,

I'm trying access a SQL Server 2000 that have a Japanese database 
(Unfortunately I have no choice), and before (old machine in 32 bit) was 
running without problem, but when I try to install in a 64 bit with the same 
configuration I have the "???" response in the field.
Checking both machines with isql and tsql select the query, both returned the 
field with character correct.

What steps will reproduce the problem?
#!/usr/bin/python
import pyodbc

server=''
port=1433
user = ''
password = ''
database = ''
driver = 'FreeTDS'

con_string = r'DRIVER={%s};SERVER=%s;PORT=%s;UID=%s;PWD=%s;DATABASE=%s;' % 
(driver, server, port, user, password, database)
cnxn = pyodbc.connect(con_string)
cursor = cnxn.cursor()
cursor.execute("""SELECT TOP 1 field FROM T_table""")
row = cursor.fetchone()
print row[0]
print row[0].decode('sjis')

What is the expected output? What do you see instead?
Expected an string with kanji and return ???

What version of the product are you using? On what operating system?
Database [ MS Server Enterpise Edition ] in Windows NT
- Version: 8.00.2040
- Language: Japanese
- Server Collation: Japanese_CI_AS

Field - Varchar(1000) - Collation: Japanese_CI_AS

[odbcinst.ini]
Working 
[FreeTDS]
Description=FreeTDS driver
Driver=/usr/lib/odbc/libtdsodbc.so

Not working
[FreeTDS]
Description = TDS driver (Sybase/MS SQL)
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so

Working
Ubuntu 10.04.4 LTS
Linux 2.6.32-24-generic-pae i686
Python 2.6.5
pyodbc 3.0.3
unixodbc (2.2.11-21) unstable; urgency=low (Checked in the 
/usr/share/doc/unixodbc/changelog.Debian.gz)
freetds v0.82

Not Working
Ubuntu 12.04.1 LTS
Linux 3.2.0-33-generic x86_64
Python 2.7.3
pyodbc 3.0.6
unixodbc (2.2.14p2-5ubuntu3) precise; urgency=low (Checked in the 
/usr/share/doc/unixodbc/changelog.Debian.gz)
freetds v0.91

Please provide any additional information below.

ODBC Trace (Remove the "equal part" ignoring memory value)
==========================================================
Version 3.0.3 - 32 bits
[ODBC][14039][SQLGetData.c][470]
                Exit:[SQL_SUCCESS]
                        Buffer = [(Late) <8e><84><97>p<82>Ì<82>½<82>ß; (Undertime) <91><81><91>Þ<81>F<81>@<8e>q<8b><9f><97>p<8e><96><82>Ì<82>½<82>ß; (Time Shift) GW]
                        Strlen Or Ind = 0xbfbd63f8 -> 70
[ODBC][14039][SQLGetData.c][224]
                Entry:
                        Column Number = 18
                        Target Type = 4 SQL_INTEGER
                        Buffer Length = 4
[ODBC][14039][SQLGetData.c][470]
                Exit:[SQL_SUCCESS]
                        Buffer = [21]
                        Strlen Or Ind = 0xbfbd6938 -> 4
[ODBC][14039][SQLGetData.c][224]
                Entry:
                        Column Number = 19
                        Target Type = 1 SQL_CHAR
                        Buffer Length = 1024
[ODBC][14039][SQLGetData.c][470]
                Exit:[SQL_SUCCESS]
                        Buffer = [DR]
                        Strlen Or Ind = 0xbfbd63f8 -> 2
[ODBC][14039][SQLGetData.c][224]
                Entry:
                        Column Number = 20
                        Target Type = 1 SQL_CHAR
                        Buffer Length = 1024
[ODBC][14039][SQLGetData.c][470]
                Exit:[SQL_SUCCESS]
                        Buffer = [DR]
                        Strlen Or Ind = 0xbfbd63f8 -> 2
[ODBC][14039][SQLGetData.c][224]
                Entry:
                        Column Number = 21
                        Target Type = 4 SQL_INTEGER
                        Buffer Length = 4
... (Back to the same)
[ODBC][14039][SQLGetInfo.c][214]
                Entry:
                        Info Type = SQL_CURSOR_ROLLBACK_BEHAVIOR (24)
                        Buffer Length = 8

Version 3.0.6 - 64 bits
[ODBC][13149][1369881310.045233][SQLGetData.c][497]
                Exit:[SQL_SUCCESS_WITH_INFO]
                        Buffer = [(Late) ?????; (Undertime) ???????????; (Time Shift) GW]
                        Strlen Or Ind = 0x7fff6937ff78 -> 54
                DIAG [42000] [FreeTDS][SQL Server]Some character(s) could not be converted into client's character set.  Unconverted bytes were changed to question marks ('?')
[ODBC][13149][1369881310.045267][SQLGetData.c][233]
                Entry:
                        Column Number = 17
                        Target Type = 1 SQL_CHAR
                        Buffer Length = 971
[ODBC][13149][1369881310.045279][SQLGetData.c][497]
                Exit:[SQL_NO_DATA]
                        Buffer = [W]
                        Strlen Or Ind = 0x7fff6937ff78 -> 0
[ODBC][13149][1369881310.045291][SQLGetData.c][233]
                Entry:
                        Column Number = 18
                        Target Type = 4 SQL_INTEGER
                        Buffer Length = 4
[ODBC][13149][1369881310.045305][SQLGetData.c][497]
                Exit:[SQL_SUCCESS]
                        Buffer = [21]
                        Strlen Or Ind = 0x7fff693804d8 -> 4
[ODBC][13149][1369881310.045316][SQLGetData.c][233]
                Entry:
                        Column Number = 19
                        Target Type = 1 SQL_CHAR
                        Buffer Length = 1024
[ODBC][13149][1369881310.045328][SQLGetData.c][497]
                Exit:[SQL_SUCCESS]
                        Buffer = [DR]
                        Strlen Or Ind = 0x7fff6937ff78 -> 2
[ODBC][13149][1369881310.045339][SQLGetData.c][233]
                Entry:
                        Column Number = 20
                        Target Type = 1 SQL_CHAR
                        Buffer Length = 1024
[ODBC][13149][1369881310.045351][SQLGetData.c][497]
                Exit:[SQL_SUCCESS]
                        Buffer = [DR]
                        Strlen Or Ind = 0x7fff6937ff78 -> 2
[ODBC][13149][1369881310.045362][SQLGetData.c][233]
                Entry:
                        Column Number = 21
                        Target Type = 4 SQL_INTEGER
                        Buffer Length = 4
... (Back to the same)
[ODBC][13149][1369881310.048778][SQLGetInfo.c][546]
                Entry:
                        Info Type = SQL_CURSOR_COMMIT_BEHAVIOR (23)
                        Buffer Length = 2
[ODBC][13149][1369881310.048795][SQLGetInfo.c][608]
                Exit:[SQL_SUCCESS]
[ODBC][13149][1369881310.048811][SQLGetInfo.c][546]
                Entry:
                        Info Type = SQL_CURSOR_ROLLBACK_BEHAVIOR (24)
                        Buffer Length = 2

DIFF Only with the select only one field
268,288c268,274
< [ODBC][13437][1369883634.677105][SQLGetData.c][497]
<       Exit:[SQL_SUCCESS_WITH_INFO]                
<           Buffer = [(Late) ?????; (Undertime) ???????????; (Time Shift) GW]          

<           Strlen Or Ind = 0x7fff64081b48 -> 54
<       DIAG [42000] [FreeTDS][SQL Server]Some character(s) could not be converted 
into client's character set.  Unconverted bytes were changed to question marks 
('?')
< 
< [ODBC][13437][1369883634.677135][SQLGetData.c][233]
<       Entry:            
<           Statement = 0x233b600            
<           Column Number = 1            
<           Target Type = 1 SQL_CHAR            
<           Buffer Length = 971            
<           Target Value = 0x7fff64081b85            
<           StrLen Or Ind = 0x7fff64081b48
< [ODBC][13437][1369883634.677144][SQLGetData.c][497]
<       Exit:[SQL_NO_DATA]                
<           Buffer = [W]                
<           Strlen Or Ind = 0x7fff64081b48 -> 0
< [ODBC][13437][1369883634.678599][SQLFreeStmt.c][140]

---
> [ODBC][14092][SQLGetData.c][470]
>       Exit:[SQL_SUCCESS]                
>           Buffer = [(Late) Ž„—p‚Ì‚½‚ß; (Undertime) 
‘‘ށF@Žq‹Ÿ—pŽ–‚Ì‚½‚ß; (Time Shift) GW]              

>           Strlen Or Ind = 0xbfede8e8 -> 70
> [ODBC][14092][SQLFreeStmt.c][140]
306,315c292
<           Connection = 0x22c9f90            
<           Info Type = SQL_CURSOR_COMMIT_BEHAVIOR (23)            
<           Info Value = 0x22cb400            
<           Buffer Length = 2            
<           StrLen = 0x7fff640821de
< [ODBC][13437][1369883634.680639][SQLGetInfo.c][608]
<       Exit:[SQL_SUCCESS]
< [ODBC][13437][1369883634.680654][SQLGetInfo.c][546]
<       Entry:            
<           Connection = 0x22c9f90            

---
>           Connection = 0x8e5c778            
317,320c294,297
<           Info Value = 0x22cb402            
<           Buffer Length = 2            
<           StrLen = 0x7fff640821de
< [ODBC][13437][1369883634.680665][SQLGetInfo.c][608]

---
>           Info Value = 0xbfedee96            
>           Buffer Length = 8            
>           StrLen = 0xbfedee94
> [ODBC][14092][SQLGetInfo.c][528]

Original issue reported on code.google.com by tsuneo.o...@gmail.com on 30 May 2013 at 3:59

GoogleCodeExporter commented 8 years ago 
[deleted comment]
GoogleCodeExporter commented 8 years ago 
Found the way:
/etc/freetds/freetds.conf
[MS_Server]
    host = localhost
    port = 1433
    tds version = 8.0
    client charset = SHIFT-JIS

/etc/odbcinst.ini
[ODBC]
Trace = No
TraceFile = /tmp/odbc.log

[FreeTDS]
Description = TDS driver (Sybase/MS SQL)
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so

/etc/odbc.ini
[ODBC Data Sources]
ODBCNAME = Microsoft SQL Server

[MS_SQL_CONNECTION]
Description = ODBC connection via FreeTDS
Driver = FreeTDS
Servername = MS_Server
Database = db_name

[Default]
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so

---------------

#!/usr/bin/python
import pyodbc

user = ''
password = ''
dsn = 'MS_SQL_CONNECTION'

con_string = r'DSN=%s;UID=%s;PWD=%s;' % (dsn, user, password)
cnxn = pyodbc.connect(con_string)
cursor = cnxn.cursor()
cursor.execute("""SELECT TOP 1 field FROM T_table""")
row = cursor.fetchone()
print row[0]
print row[0].decode('sjis')

Original comment by tsuneo.o...@gmail.com on 4 Jun 2013 at 1:51