search

google-code-export / pymssql

Automatically exported from code.google.com/p/pymssql
GNU Lesser General Public License v2.1
0 stars 0 forks source link

Threaded Tests caused crash #47

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
testErrorSprocThreadedUse (tests.test_threaded.ThreadedTests) ... SKIP
testErrorThreadedUse (tests.test_threaded.ThreadedTests) ... ok
testThreadedUse (tests.test_threaded.ThreadedTests) ... *** glibc detected *** 
/home/rsyring/.virtualenvs/pymssql/bin/python: double free or corruption 
(!prev): 0x0000000002a129a0 ***
======= Backtrace: =========
/lib/libc.so.6(+0x775b6)[0x7faf3d57c5b6]
/lib/libc.so.6(cfree+0x73)[0x7faf3d582e83]
/home/rsyring/.virtualenvs/pymssql/bin/python[0x44fc6f]
/home/rsyring/.virtualenvs/pymssql/bin/python(PyObject_GenericGetAttr+0x140)[0x4
53d10]
/home/rsyring/.virtualenvs/pymssql/bin/python(PyEval_EvalFrameEx+0x2b75)[0x4a566
5]
/home/rsyring/.virtualenvs/pymssql/bin/python(PyEval_EvalFrameEx+0x5a60)[0x4a855
0]
======= Memory map: ========
00400000-0061c000 r-xp 00000000 fc:00 3039687                            
/home/rsyring/.virtualenvs/pymssql/bin/python
0081b000-0081c000 r--p 0021b000 fc:00 3039687                            
/home/rsyring/.virtualenvs/pymssql/bin/python
0081c000-0087e000 rw-p 0021c000 fc:00 3039687                            
/home/rsyring/.virtualenvs/pymssql/bin/python
0087e000-0088d000 rw-p 00000000 00:00 0 
02338000-03441000 rw-p 00000000 00:00 0                                  [heap]
7faf34000000-7faf34021000 rw-p 00000000 00:00 0 
7faf34021000-7faf38000000 ---p 00000000 00:00 0 
7faf393de000-7faf393f4000 r-xp 00000000 fc:00 603                        
/lib/libgcc_s.so.1
7faf393f4000-7faf395f3000 ---p 00016000 fc:00 603                        
/lib/libgcc_s.so.1
7faf395f3000-7faf395f4000 r--p 00015000 fc:00 603                        
/lib/libgcc_s.so.1
7faf395f4000-7faf395f5000 rw-p 00016000 fc:00 603                        
/lib/libgcc_s.so.1
7faf395f5000-7faf395f6000 ---p 00000000 00:00 0 
7faf395f6000-7faf39df6000 rw-p 00000000 00:00 0 
7faf39df6000-7faf39df7000 ---p 00000000 00:00 0 
7faf39df7000-7faf3a5f7000 rw-p 00000000 00:00 0 
7faf3a5f7000-7faf3a5f9000 r-xp 00000000 fc:00 14450                      
/lib/libnss_mdns4.so.2
7faf3a5f9000-7faf3a7f8000 ---p 00002000 fc:00 14450                      
/lib/libnss_mdns4.so.2
7faf3a7f8000-7faf3a7f9000 r--p 00001000 fc:00 14450                      
/lib/libnss_mdns4.so.2
7faf3a7f9000-7faf3a7fa000 rw-p 00002000 fc:00 14450                      
/lib/libnss_mdns4.so.2
7faf3a7fa000-7faf3a810000 r-xp 00000000 fc:00 20688                      
/lib/libresolv-2.11.1.so
7faf3a810000-7faf3aa0f000 ---p 00016000 fc:00 20688                      
/lib/libresolv-2.11.1.so
7faf3aa0f000-7faf3aa10000 r--p 00015000 fc:00 20688                      
/lib/libresolv-2.11.1.so
7faf3aa10000-7faf3aa11000 rw-p 00016000 fc:00 20688                      
/lib/libresolv-2.11.1.so
7faf3aa11000-7faf3aa13000 rw-p 00000000 00:00 0 
7faf3aa13000-7faf3aa18000 r-xp 00000000 fc:00 20681                      
/lib/libnss_dns-2.11.1.so
7faf3aa18000-7faf3ac17000 ---p 00005000 fc:00 20681                      
/lib/libnss_dns-2.11.1.so
7faf3ac17000-7faf3ac18000 r--p 00004000 fc:00 20681                      
/lib/libnss_dns-2.11.1.so
7faf3ac18000-7faf3ac19000 rw-p 00005000 fc:00 20681                      
/lib/libnss_dns-2.11.1.so
7faf3ac19000-7faf3ac1b000 r-xp 00000000 fc:00 14451                      
/lib/libnss_mdns4_minimal.so.2
7faf3ac1b000-7faf3ae1a000 ---p 00002000 fc:00 14451                      
/lib/libnss_mdns4_minimal.so.2
7faf3ae1a000-7faf3ae1b000 r--p 00001000 fc:00 14451                      
/lib/libnss_mdns4_minimal.so.2
7faf3ae1b000-7faf3ae1c000 rw-p 00002000 fc:00 14451                      
/lib/libnss_mdns4_minimal.so.2
7faf3ae1c000-7faf3ae1e000 r-xp 00000000 fc:00 1050680                    
/usr/lib/gconv/CP1252.so
7faf3ae1e000-7faf3b01d000 ---p 00002000 fc:00 1050680                    
/usr/lib/gconv/CP1252.so
7faf3b01d000-7faf3b01e000 r--p 00001000 fc:00 1050680                    
/usr/lib/gconv/CP1252.so
7faf3b01e000-7faf3b01f000 rw-p 00002000 fc:00 1050680                    
/usr/lib/gconv/CP1252.so
7faf3b01f000-7faf3b021000 r-xp 00000000 fc:00 1056540                    
/usr/lib/gconv/ISO8859-1.so
7faf3b021000-7faf3b220000 ---p 00002000 fc:00 1056540                    
/usr/lib/gconv/ISO8859-1.so
7faf3b220000-7faf3b221000 r--p 00001000 fc:00 1056540                    
/usr/lib/gconv/ISO8859-1.so
7faf3b221000-7faf3b222000 rw-p 00002000 fc:00 1056540                    
/usr/lib/gconv/ISO8859-1.so
7faf3b222000-7faf3b22e000 r-xp 00000000 fc:00 20682                      
/lib/libnss_files-2.11.1.so
7faf3b22e000-7faf3b42d000 ---p 0000c000 fc:00 20682                      
/lib/libnss_files-2.11.1.so
7faf3b42d000-7faf3b42e000 r--p 0000b000 fc:00 20682                      
/lib/libnss_files-2.11.1.so
7faf3b42e000-7faf3b42f000 rw-p 0000c000 fc:00 20682                      
/lib/libnss_files-2.11.1.so
7faf3b42f000-7faf3b439000 r-xp 00000000 fc:00 20684                      
/lib/libnss_nis-2.11.1.so
7faf3b439000-7faf3b638000 ---p 0000a000 fc:00 20684                      
/lib/libnss_nis-2.11.1.so
7faf3b638000-7faf3b639000 r--p 00009000 fc:00 20684                      
/lib/libnss_nis-2.11.1.so
7faf3b639000-7faf3b63a000 rw-p 0000a000 fc:00 20684                      
/lib/libnss_nis-2.11.1.so
7faf3b63a000-7faf3b651000 r-xp 00000000 fc:00 20677                      
/lib/libnsl-2.11.1.so
7faf3b651000-7faf3b850000 ---p 00017000 fc:00 20677                      
/lib/libnsl-2.11.1.so
7faf3b850000-7faf3b851000 r--p 00016000 fc:00 20677                      
/lib/libnsl-2.11.1.so
7faf3b851000-7faf3b852000 rw-p 00017000 fc:00 20677                      
/lib/libnsl-2.11.1.so
7faf3b852000-7faf3b854000 rw-p 00000000 00:00 0 
7faf3b854000-7faf3b85c000 r-xp 00000000 fc:00 20678                      
/lib/libnss_compat-2.11.1.so
7faf3b85c000-7faf3ba5b000 ---p 00008000 fc:00 20678                      
/lib/libnss_compat-2.11.1.so
7faf3ba5b000-7faf3ba5c000 r--p 00007000 fc:00 20678                      
/lib/libnss_compat-2.11.1.so
7faf3ba5c000-7faf3ba5d000 rw-p 00008000 fc:00 20678                      
/lib/libnss_compat-2.11.1.so
7faf3ba5d000-7faf3ba79000 r-xp 00000000 fc:00 2892531                    
/home/rsyring/dev/python/pymssql/pymssql-src/pymssql.so
7faf3ba79000-7faf3bc78000 ---p 0001c000 fc:00 2892531                    
/home/rsyring/dev/python/pymssql/pymssql-src/pymssql.so
7faf3bc78000-7faf3bc79000 r--p 0001b000 fc:00 2892531                    
/home/rsyring/dev/python/pymssql/pymssql-src/pymssql.so
7faf3bc79000-7faf3bc7d000 rw-p 0001c000 fc:00 2892531                    
/home/rsyring/dev/python/pymssql/pymssql-src/pymssql.so
7faf3bc7d000-7faf3bc7e000 rw-p 00000000 00:00 0 
7faf3bc7e000-7faf3bc84000 r-xp 00000000 fc:00 789208                     
/usr/lib/python2.6/dist-packages/zope/interface/_zope_interface_coptimizations.s
oAborted

But i was not able to reproduce a second time.

Original issue reported on code.google.com by rsyr...@gmail.com on 16 Apr 2011 at 7:25

GoogleCodeExporter commented 9 years ago 
I take it back, I saw it again:

*** glibc detected *** /home/rsyring/.virtualenvs/pymssql/bin/python: malloc(): 
memory corruption: 0x00007fe7ec000fc0 ***
======= Backtrace: =========
/lib/libc.so.6(+0x775b6)[0x7fe7f59e55b6]
/lib/libc.so.6(+0x7b6d8)[0x7fe7f59e96d8]
/lib/libc.so.6(__libc_malloc+0x6e)[0x7fe7f59ea58e]
/usr/local/lib/libsybdb.so.5(+0x21ede)[0x7fe7f4b54ede]
/usr/local/lib/libsybdb.so.5(+0x23927)[0x7fe7f4b56927]
/usr/local/lib/libsybdb.so.5(+0x241d2)[0x7fe7f4b571d2]
/usr/local/lib/libsybdb.so.5(+0x257f4)[0x7fe7f4b587f4]
/usr/local/lib/libsybdb.so.5(dbsqlok+0xa1)[0x7fe7f4b43e91]
/home/rsyring/dev/python/pymssql/pymssql-src/_mssql.so(+0x1151a)[0x7fe7f4dab51a]
/home/rsyring/dev/python/pymssql/pymssql-src/_mssql.so(+0x93e7)[0x7fe7f4da33e7]
/home/rsyring/dev/python/pymssql/pymssql-src/_mssql.so(+0xa677)[0x7fe7f4da4677]
/home/rsyring/.virtualenvs/pymssql/bin/python(PyEval_EvalFrameEx+0x516e)[0x4a7c5
e]
/home/rsyring/.virtualenvs/pymssql/bin/python(PyEval_EvalFrameEx+0x5a60)[0x4a855
0]
======= Memory map: ========
00400000-0061c000 r-xp 00000000 fc:00 3039687                            
/home/rsyring/.virtualenvs/pymssql/bin/python
0081b000-0081c000 r--p 0021b000 fc:00 3039687                            
/home/rsyring/.virtualenvs/pymssql/bin/python
0081c000-0087e000 rw-p 0021c000 fc:00 3039687                            
/home/rsyring/.virtualenvs/pymssql/bin/python
0087e000-0088d000 rw-p 00000000 00:00 0 
011fb000-02308000 rw-p 00000000 00:00 0                                  [heap]
7fe7ec000000-7fe7ec021000 rw-p 00000000 00:00 0 
7fe7ec021000-7fe7f0000000 ---p 00000000 00:00 0 
7fe7f1847000-7fe7f185d000 r-xp 00000000 fc:00 603                        
/lib/libgcc_s.so.1
7fe7f185d000-7fe7f1a5c000 ---p 00016000 fc:00 603                        
/lib/libgcc_s.so.1
7fe7f1a5c000-7fe7f1a5d000 r--p 00015000 fc:00 603                        
/lib/libgcc_s.so.1
7fe7f1a5d000-7fe7f1a5e000 rw-p 00016000 fc:00 603                        
/lib/libgcc_s.so.1
7fe7f1a5e000-7fe7f1a5f000 ---p 00000000 00:00 0 
7fe7f1a5f000-7fe7f225f000 rw-p 00000000 00:00 0 
7fe7f225f000-7fe7f2260000 ---p 00000000 00:00 0 
7fe7f2260000-7fe7f2a60000 rw-p 00000000 00:00 0 
7fe7f2a60000-7fe7f2a62000 r-xp 00000000 fc:00 14450                      
/lib/libnss_mdns4.so.2
7fe7f2a62000-7fe7f2c61000 ---p 00002000 fc:00 14450                      
/lib/libnss_mdns4.so.2
7fe7f2c61000-7fe7f2c62000 r--p 00001000 fc:00 14450                      
/lib/libnss_mdns4.so.2
7fe7f2c62000-7fe7f2c63000 rw-p 00002000 fc:00 14450                      
/lib/libnss_mdns4.so.2
7fe7f2c63000-7fe7f2c79000 r-xp 00000000 fc:00 20688                      
/lib/libresolv-2.11.1.so
7fe7f2c79000-7fe7f2e78000 ---p 00016000 fc:00 20688                      
/lib/libresolv-2.11.1.so
7fe7f2e78000-7fe7f2e79000 r--p 00015000 fc:00 20688                      
/lib/libresolv-2.11.1.so
7fe7f2e79000-7fe7f2e7a000 rw-p 00016000 fc:00 20688                      
/lib/libresolv-2.11.1.so
7fe7f2e7a000-7fe7f2e7c000 rw-p 00000000 00:00 0 
7fe7f2e7c000-7fe7f2e81000 r-xp 00000000 fc:00 20681                      
/lib/libnss_dns-2.11.1.so
7fe7f2e81000-7fe7f3080000 ---p 00005000 fc:00 20681                      
/lib/libnss_dns-2.11.1.so
7fe7f3080000-7fe7f3081000 r--p 00004000 fc:00 20681                      
/lib/libnss_dns-2.11.1.so
7fe7f3081000-7fe7f3082000 rw-p 00005000 fc:00 20681                      
/lib/libnss_dns-2.11.1.so
7fe7f3082000-7fe7f3084000 r-xp 00000000 fc:00 14451                      
/lib/libnss_mdns4_minimal.so.2
7fe7f3084000-7fe7f3283000 ---p 00002000 fc:00 14451                      
/lib/libnss_mdns4_minimal.so.2
7fe7f3283000-7fe7f3284000 r--p 00001000 fc:00 14451                      
/lib/libnss_mdns4_minimal.so.2
7fe7f3284000-7fe7f3285000 rw-p 00002000 fc:00 14451                      
/lib/libnss_mdns4_minimal.so.2
7fe7f3285000-7fe7f3287000 r-xp 00000000 fc:00 1050680                    
/usr/lib/gconv/CP1252.so
7fe7f3287000-7fe7f3486000 ---p 00002000 fc:00 1050680                    
/usr/lib/gconv/CP1252.so
7fe7f3486000-7fe7f3487000 r--p 00001000 fc:00 1050680                    
/usr/lib/gconv/CP1252.so
7fe7f3487000-7fe7f3488000 rw-p 00002000 fc:00 1050680                    
/usr/lib/gconv/CP1252.so
7fe7f3488000-7fe7f348a000 r-xp 00000000 fc:00 1056540                    
/usr/lib/gconv/ISO8859-1.so
7fe7f348a000-7fe7f3689000 ---p 00002000 fc:00 1056540                    
/usr/lib/gconv/ISO8859-1.so
7fe7f3689000-7fe7f368a000 r--p 00001000 fc:00 1056540                    
/usr/lib/gconv/ISO8859-1.so
7fe7f368a000-7fe7f368b000 rw-p 00002000 fc:00 1056540                    
/usr/lib/gconv/ISO8859-1.so
7fe7f368b000-7fe7f3697000 r-xp 00000000 fc:00 20682                      
/lib/libnss_files-2.11.1.so
7fe7f3697000-7fe7f3896000 ---p 0000c000 fc:00 20682                      
/lib/libnss_files-2.11.1.so
7fe7f3896000-7fe7f3897000 r--p 0000b000 fc:00 20682                      
/lib/libnss_files-2.11.1.so
7fe7f3897000-7fe7f3898000 rw-p 0000c000 fc:00 20682                      
/lib/libnss_files-2.11.1.so
7fe7f3898000-7fe7f38a2000 r-xp 00000000 fc:00 20684                      
/lib/libnss_nis-2.11.1.so
7fe7f38a2000-7fe7f3aa1000 ---p 0000a000 fc:00 20684                      
/lib/libnss_nis-2.11.1.so
7fe7f3aa1000-7fe7f3aa2000 r--p 00009000 fc:00 20684                      
/lib/libnss_nis-2.11.1.so
7fe7f3aa2000-7fe7f3aa3000 rw-p 0000a000 fc:00 20684                      
/lib/libnss_nis-2.11.1.so
7fe7f3aa3000-7fe7f3aba000 r-xp 00000000 fc:00 20677                      
/lib/libnsl-2.11.1.so
7fe7f3aba000-7fe7f3cb9000 ---p 00017000 fc:00 20677                      
/lib/libnsl-2.11.1.so
7fe7f3cb9000-7fe7f3cba000 r--p 00016000 fc:00 20677                      
/lib/libnsl-2.11.1.so
7fe7f3cba000-7fe7f3cbb000 rw-p 00017000 fc:00 20677                      
/lib/libnsl-2.11.1.so
7fe7f3cbb000-7fe7f3cbd000 rw-p 00000000 00:00 0 
7fe7f3cbd000-7fe7f3cc5000 r-xp 00000000 fc:00 20678                      
/lib/libnss_compat-2.11.1.so
7fe7f3cc5000-7fe7f3ec4000 ---p 00008000 fc:00 20678                      
/lib/libnss_compat-2.11.1.so
7fe7f3ec4000-7fe7f3ec5000 r--p 00007000 fc:00 20678                      
/lib/libnss_compat-2.11.1.so
7fe7f3ec5000-7fe7f3ec6000 rw-p 00008000 fc:00 20678                      
/lib/libnss_compat-2.11.1.so
7fe7f3ec6000-7fe7f3ee2000 r-xp 00000000 fc:00 2892531                    
/home/rsyring/dev/python/pymssql/pymssql-src/pymssql.so
7fe7f3ee2000-7fe7f40e1000 ---p 0001c000 fc:00 2892531                    
/home/rsyring/dev/python/pymssql/pymssql-src/pymssql.soAborted

I added a SkipTest to the threading tests module for now.

Original comment by rsyr...@gmail.com on 16 Apr 2011 at 7:46

GoogleCodeExporter commented 9 years ago 
I had seen similar problems (python.exe interpreter crash with a illegal memory 
access error message) when testing pymssql tip together with:

* Official Python 2.7.2 win32 binaries from python.org.
* FreeTDS from the 0.91 development CVS branch compiled as statil libraries 
with the same compiler as official Python binaries (MS VC 2008) (see issue #61)
* Cython 0.14.1 (from http://www.lfd.uci.edu/~gohlke/pythonlibs/#cython)

All under Windows 7.

Now, I've updated my working enviroment to:

* FreeTDS to release 0.91 (see issue #64)
* Cython to release 0.15

and the problem doesn't happen anymore. I suspect it's because the Cython 
upgrade.

Can you re-test things?

Original comment by cra...@gmail.com on 6 Sep 2011 at 2:47

GoogleCodeExporter commented 9 years ago 
Thanks for the report.  Thats great news!  I will re-test.

Original comment by rsyr...@gmail.com on 6 Sep 2011 at 3:06

GoogleCodeExporter commented 9 years ago 
Unfortunately, I still see segfaults when running tests/test_threaded.py with:

python 2.6/2.7
linux x64
Cython 0.15.1
FreeTDS 0.91

Original comment by rsyr...@gmail.com on 18 Oct 2011 at 3:58

GoogleCodeExporter commented 9 years ago 
Interestingly, I ran the threaded tests on a 32 bit system, 3 times each for 
both PY 2.6/2.7 and never saw a segfault.  I get them almost immediately on x64.

Still no sure where that leaves us, but I guess its good to know.

Original comment by rsyr...@gmail.com on 18 Oct 2011 at 4:35

GoogleCodeExporter commented 9 years ago 
I've got a few sporadic interpreter crashes using this driver.

From examining the code, it looks like it affects every version I've looked at, 
from 1.0.2 in debian squeeze to the current hg version, they all have the same 
problem:  they don't lock the GIL during the msg_handler or err_handler 
functions.

The 1.0.2 branch in mercurial on google code (which is different from the 1.0.2 
source from debian), actually goes through the trouble of holding the lock when 
doing select db as a work around:  (Selecting a database may always cause a 
msg_handler to be invoked)
http://code.google.com/p/pymssql/source/browse/mssqldbmodule.c?name=1.0.2#287

This is a subtle and dangerous bug.  If you use threads anywhere in your python 
script, you risk interpreter crash whenever the driver throws an error or 
message -- even if you only have a single mssql connection.

The attached patch hasn't been tested, or even attempted to compile, it's just 
an example of what needs to happen to fix.

Original comment by rol...@gmail.com on 26 Jan 2012 at 2:31

Attachments:

GoogleCodeExporter commented 9 years ago 
On single CPU machine the threaded tests work fine for me without the patch.  
Testing on a box with 24 hardware threads, it crashes real quick without the 
patch.

Tested the lock_msg_err_handler.patch works great for me, no crash:

$ nosetests test_threaded.py
S..
----------------------------------------------------------------------
Ran 3 tests in 28.693s

OK (SKIP=1)
(Still skipping testErrorSprocThreadedUse, I don't know what's up with that 
one).

A patch for 1.0.2 / Debian Squeeze looks a lot more complicated.  Cython was a 
definite win here.  It's nice to have GIL maintenance as a language feature.

Original comment by rol...@gmail.com on 26 Jan 2012 at 10:19

GoogleCodeExporter commented 9 years ago 
just pushed the change in the patch.

Original comment by rsyr...@gmail.com on 3 Apr 2012 at 3:49

GoogleCodeExporter commented 9 years ago 
I was wondering if older versions of pymssql are to be fixed too (previous than 
2.0.0b1) ?
I'd like to use pymssql on a production site and i'd like to avoid development 
versions, unless 2.0.0b1 is closer to a release candidate than a development 
product.

Regards

Original comment by julien.s...@gmail.com on 27 Jun 2012 at 9:17

GoogleCodeExporter commented 9 years ago 
From google code home page:

I don't think I would recommend pymssql (any version) for a production 
site unless you have thorough tests or QA environment that can run 
through the pymssql functionality you use.  I use it in a development 
environment and have not run into any unknown bugs or any bugs that 
prevent me from doing the work I need to do with it.  But how I use it 
and how you use it may be very different things.

You can also checkout pyodbc w/ the FreeTDS ODBC driver or the Microsoft 
Linux ODBC driver.  pyodbc also works with MSSQL drivers on Windows.

Original comment by rsyr...@gmail.com on 27 Jun 2012 at 12:32