linux/freetds segfault with unicode strings

[GoogleCodeExporter]

Platform:
---------

Ubuntu 10.04 x64
Python 2.6.5
PyODBC 2.1.8 & also tried current git version
FreeTDS 0.82-1build6 (distro version) and also tried custom CVS build

Reproduce:
----------

Run the attached script on similar platform

What is the expected output?
----------------------------

'would commit, but segfault happens first'

What do you see instead?
------------------------

*** glibc detected *** /path/to/python/or/script: free(): invalid next size 
(fast): 0x0000000003e605b0 ***

or 

*** glibc detected *** /path/to/python/or/script: malloc(): memory corruption: 
0x00000000026c26e0 ***

or rarely

python: malloc.c:3096: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) 
&((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) 
&& old_size == 0) || ((unsigned long) (old_size) >= (unsigned 
long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * 
(sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 
0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.

Other Info:
-----------

Note the comments in the attached file.  The other combinations of string 
length run fine.

Let me know if there is anything else I can do to help track this down.

Original issue reported on code.google.com by rsyr...@gmail.com on 7 Apr 2011 at 6:43

Attachments:

• test_pyodbc.py

[GoogleCodeExporter]

TDS Dump when running the script is attached.

Original comment by rsyr...@gmail.com on 7 Apr 2011 at 6:46

Attachments:

• tdsdump.log

[GoogleCodeExporter]

ok, a little more info and a correction.  When running the test script against 
the distro's version of FreeTDS, I do not get a segfault.  Instead, I get the 
error:

pyodbc.Error: ('HY004', '[HY004] [FreeTDS][SQL Server]Invalid data type (0) 
(SQLBindParameter)

When I run the pyodbc tests (sqlservertests.py) against 0.82, then I get a lot 
of failures (output attached).  When I run them against my compiled version of 
freetds, they segfault.

Original comment by rsyr...@gmail.com on 8 Apr 2011 at 4:19

Attachments:

• test_errors.txt

[GoogleCodeExporter]

(SQLBindParameter) error was fixed by using a freshly compiled version of 
FreeTDS and setting ClientCharset=UTF-8 in the connection string.  Doing that 
makes the test script run without error.

However, pyodbc tests still segfault and I still get segfault problems when 
running unit tests on a web application I built using SQLAlchemy.

Original comment by rsyr...@gmail.com on 8 Apr 2011 at 6:05

[GoogleCodeExporter]

Sorry for all the comments, I am continuing to try to figure this one out.  

I was able to prevent my application's unit tests from crashing by changing all 
my SQLAlchemy columns from String() to Unicode().  That lead me to be able to 
come up with a test script that reproduces the problem (attached) based on 
SQLAlchemy.  

In my attempt to translate that code to pure pyodbc (the test script previously 
attached), the error no longer shows up.  However, I'm pretty confident the bug 
is in pyodbc or freetds since pure python code shouldn't be able to crash 
python.

FreeTDS guys suggested I run valgrind.  So I did that:

valgrind --tool=memcheck -v --suppressions=valgrind-python.supp python 
test_freetds_unicode.py 2>valgrind-out.txt

based on the files here:

http://svn.python.org/projects/python/trunk/Misc/README.valgrind
http://svn.python.org/projects/python/trunk/Misc/valgrind-python.supp (I 
uncommented the parts they recommend uncommenting if you don't want to 
re-compile python)

However, when I run with valgrind, I don't get a segfault anymore!  Valgrind 
output is attached anyway, not sure how helpful it will be.

Let me know if there is anything else I can to help.

Original comment by rsyr...@gmail.com on 8 Apr 2011 at 3:58

Attachments:

• test_freetds_unicode.py
• valgrind-out.txt

[GoogleCodeExporter]

This is worrisome: Use of uninitialised value of size 8

However, it seems to show up a lot.  Can you run valgrind on a simple Python 
script without pyodbc to see if it also generates these errors.  It might be an 
error in the Python runtime:

  class X(object): pass
  x = X()
  x.test = 1
  del x

Original comment by mkleehammer on 20 May 2011 at 7:36

• Changed state: Investigating

[GoogleCodeExporter]

Im getting this error when using Easysoft, not FreeTDS, drivers.   Im getting 
the ***glibc detected*** behavior on a fedora machine as well as an ubuntu 
machine - on a remote machine being tested by a colleague, he is getting the  
"Invalid data type (0) (SQLBindParameter)" error.   I've never had this problem 
with FreeTDS.

Original comment by zzz...@gmail.com on 6 Jun 2011 at 7:36

[GoogleCodeExporter]

I believe there is a bug in sqlwchar.cpp at line 154.  I think it should be len 
+ 1 because sqlwchar_copy() includes the trailing null.

Also, the freeTDS 0.91 release seems to fix a lot of unicode related issues.

Original comment by davidp.r...@gmail.com on 7 Jun 2011 at 1:27

[GoogleCodeExporter]

I am having this issue as well, and running original reporter's test_pyodbc.py 
script results in same errors. Error occurs on two different systems:

1. Gentoo x64, python 2.6.6, unixodbc 2.3.0, freetds 0.83dev (from 2011-01-25), 
pyodbc 2.1.8
2. Gentoo x64, python 2.7.1, unixodbc 2.3.0, freetds 0.91, pyodbc 2.1.11

Original comment by asha...@chumpland.org on 5 Oct 2011 at 3:28

[GoogleCodeExporter]

Thank you, davidp.r...@gmail.com (comment #7)!

Your suggestion with len in sqlwchar.cpp at line 154 helped me (see attached 
patch).

OpenSUSE 11.1, Python 2.6.2, pyodbc 2.1.11, freetds 0.91.

Original comment by alex.s.y...@gmail.com on 26 Oct 2011 at 12:04

Attachments:

• pyodbc_sqlwchar.patch

[GoogleCodeExporter]

I am very grateful for you guys finding this.  Fixed in 2.1.13-beta01 and 
py3-3.0.1-beta06.

Original comment by mkleehammer on 27 Oct 2011 at 6:44

• Changed state: Complete

[GoogleCodeExporter]

Thank you for that patch. I hope to see it soon in stable version.

Original comment by mike.dob...@gmail.com on 5 Nov 2011 at 11:32