Describe what extension authors are supposed to do

[GoogleCodeExporter]

Please can you document in the wiki exactly what an extension developer is 
supposed to do to avoid the issues with C extensions, modwsgi and the 
simplified GIL api.

I'd be delighted to update my extension, but do not know how things should be 
changed.  The extensions supports Python 2.3 all the way to 3.2.

Original issue reported on code.google.com by rogerbinns on 3 Dec 2010 at 6:29

[GoogleCodeExporter]

Is your C extension publicly available so can look at what you are doing now?

Original comment by Graham.Dumpleton@gmail.com on 4 Dec 2010 at 12:20

• Added labels: Type-Other
• Removed labels: Type-Defect

[GoogleCodeExporter]

Yes.  The home page is here:

  http://code.google.com/p/apsw/

The code is spread across a few .c files but they are all #included into apsw.c 
so that it forms a single compilation unit.

When calling into SQLite (releasing the GIL), everything goes through the 
macros at the top of this file:

  http://code.google.com/p/apsw/source/browse/src/util.c

When calling back into Python code (ie acquiring the GIL) then there are 
several callbacks, most in this file:

  http://code.google.com/p/apsw/source/browse/src/connection.c

I believe that my usage of the simplified APIs is "textbook".

Original comment by rogerbinns on 4 Dec 2010 at 1:09

[GoogleCodeExporter]

pymonogo is an open source module I use in a separate project.  They also have 
an issue with modwsgi.  However in their case they have pure Python code 
(slower) to fall back to.

  http://api.mongodb.org/python/1.9%2B/faq.html#does-pymongo-work-with-mod-wsgi

Looking through their code I don't see any usage of the GIL at all.  Knowing 
what us extension authors are supposed to do would be very helpful.

Original comment by rogerbinns on 4 Dec 2010 at 1:17

[GoogleCodeExporter]

The callbacks is what will cause a hang in Python sub interpreter. Simple thing 
is to use:

  WSGIApplicationGroup %{GLOBAL}

to force use of main interpreter for any WSGI application using your module.

I would have to look more closely at your code to suggest how you otherwise get 
around that. May require any user parameters you register with sqlite callback 
mechanism to also reference thread interpreter state, so it can switch back to 
that thread context rather than use PyGilState_???() functions.

Original comment by Graham.Dumpleton@gmail.com on 4 Dec 2010 at 1:23

[GoogleCodeExporter]

The pymongo case may have been incorrect use of C static variables to record 
Python objects, which those objects then being incorrectly used in a different 
interpreter. One isn't supposed to mix Python objects from one sub interpreter 
with another sub interpreter.

Anyway, will stick such documentation on todo list. I may even already have an 
issue for it, can't remember.

Original comment by Graham.Dumpleton@gmail.com on 4 Dec 2010 at 1:25

[GoogleCodeExporter]

The goal is to change my code so this is a non-issue.  I have asked you several 
times before without ever getting an answer :-)

What I am looking for is a page of documentation, roughly along the lines of a 
list saying "instead of doing the usual X (eg existing GIL apis), you need to 
do Y (whatever that is)".  If I know what Y is I can change my code. In terms 
of the "thread interpreter state" I'll need to know *exactly* which apis to 
use.  I've encountered them once before and the documentation is spectacularly 
unhelpful.

For pymongo there seem to be a bunch of static PyObject pointers in 
bson/_cbsonmodule.c and I have a bunch in src/apsw.c.  

The Python doc encourages using static, and I'd expect most modules to have 
been written this way.  For example this page has 80 references to static 
although the vast majority are static methods:

  http://docs.python.org/extending/newtypes.html

ie you can assume that almost all C extension authors have been using the 
simplified GIL api and using lots of static because that is exactly what the 
Python doc tells us to do!  Show us what we need to do differently.

Original comment by rogerbinns on 4 Dec 2010 at 3:07

[GoogleCodeExporter]

Closing out documentation issues. Will pick up another time on github side when 
get enough momentum to work on documentation again.

Original comment by Graham.Dumpleton@gmail.com on 12 Nov 2014 at 10:14

• Changed state: WontFix