ibm_db.pyd is not thread-safe (with patch)

[GoogleCodeExporter]

ibm_db version 1.0.1 (latest, win32). We running some python code in
parallel - several threads, and each thread does access to DB2 database
using ibm_db adapter. This causes python.exe (win32) to crash - without
much information left behind - traceback is messed-up.

Looking into to C code of ibm_db.c I found no thread-safe blocks
(http://docs.python.org/c-api/init.html#thread-state-and-the-global-interpreter-
lock).
Therefore I patched ibm_db.c with:

   Py_BEGIN_ALLOW_THREADS;
   ...
   Py_END_ALLOW_THREADS;

for following db2 sql api calls:

   SQLConnect
   SQLExecute
   SQLFetchScroll

With this patch ibm_db.pyd doesn't cause any fatal failure problems in
threading scenario. 

In attachment one can find patch of ibm_db.c, and pyd for python 2.5 and 2.6.

Suggestion: if you could include tests for threading scenario in your
unit-test set.

Original issue reported on code.google.com by trebor74hr@gmail.com on 1 Apr 2010 at 2:22

Attachments:

• 20100401_ibm_db-threading-fix.diff
• ibm_db-2.5.pyd
• ibm_db-2.6.pyd

[GoogleCodeExporter]

Please check if any other code block/api call deserves python thread-safe block.

Original comment by trebor74hr@gmail.com on 1 Apr 2010 at 2:24

[GoogleCodeExporter]

>>>>
We running some python code in
parallel - several threads, and each thread does access to DB2 database
using ibm_db adapter. This causes python.exe (win32) to crash - without
much information left behind - traceback is messed-up.
<<<<

can you give me a sample test case  through which we can regenerate this 
problem.
Because in general python threads acquires GIL before calling Python/C API 
function.
So, above problem should not be happen.

>>>>
Looking into to C code of ibm_db.c I found no thread-safe blocks

Therefore I patched ibm_db.c with:

   Py_BEGIN_ALLOW_THREADS;
   ...
   Py_END_ALLOW_THREADS;
<<<<

By default Python handles Thread safe property for Python/C API function call,
Through acquiring GIL.

And by using “Py_BEGIN_ALLOW_THREADS” we are releasing GIL before calling 
db2 sql api call, so through instead of gaining thread safe we are allowing 
other
thread to do their job.
But through this we will get performance gain.

Original comment by rahul.pr...@in.ibm.com on 10 May 2010 at 8:25

[GoogleCodeExporter]

I'm sorry but I can't give you test case for reproducing the problem, just short
explanation: 

on some moment a thread is created which does some db 
select/delete/update/insert,
and in the "same" moment main program continues - and the logic can also do some
select/upd/del/ins. From time to time this crashes. After commenting out thread 
part,
it is ok, putting thread part alive again, it failed. After putting Py_*THREAD 
macro
around some "critical" db2 calls it stopped failing.

Original comment by trebor74hr@gmail.com on 10 May 2010 at 3:07

[GoogleCodeExporter]

I tried a lot to reproduce this problem with no luck.
Further by adding Py_*THREAD macro you are making those db calls as non 
interrupting
calls(which can't interrupt other threads). so, during these db calls other 
threads
will be able to do their works.

Are you using same connection object for both children and main thread?

Original comment by rahul.pr...@in.ibm.com on 12 May 2010 at 8:24

[GoogleCodeExporter]

to be more specific, this is done in django application, where request 
middleware
spawns new thread which does some syncing job, and then mw passes control to the
django view which can modify data in the same db. I don't know if this is the 
same
connection, don't know details how django does this (only django ORM api is 
used). My
assumption is: yes, they have same connection.

Original comment by trebor74hr@gmail.com on 17 May 2010 at 1:14

[GoogleCodeExporter]

I have created a patch for unblocking costly db calls(now these calls will not 
block
other threads execution).
please find the patch in attachment and test it in your environment. 

After using this patch let me know how it works and this is able to resolve your
problem or not. 

Depending upon your feedback i shall include this patch in our next release.

Original comment by rahul.pr...@in.ibm.com on 20 May 2010 at 10:43

Attachments:

• ibm_db.c

[GoogleCodeExporter]

Sorry for delay, currently I'm not able to test this, but will next week. Did
overview, noticed that you did a lot "ALLOW_THREADS" wraps, looks good, and 
hope this
won't decrease performance.

Original comment by trebor74hr@gmail.com on 3 Jun 2010 at 11:03

[GoogleCodeExporter]

Finally I managed to test the patched version, and good news is that problems I 
had before (python.exe crashes) are not present any more. I think you could 
include this patch in next release.

Original comment by trebor74hr@gmail.com on 29 Jun 2010 at 10:44

[GoogleCodeExporter]

Thanks robert for testing this patch. we will include this patch to our next 
release.

Original comment by rahul.pr...@in.ibm.com on 2 Jul 2010 at 6:40

[GoogleCodeExporter]

Released with ibm_db-1.0.3

Original comment by rahul.pr...@in.ibm.com on 9 Aug 2010 at 7:45

• Changed state: Verified