Segfault when cleaning up threads (with python 2.7)

[msdemlei]

Knowing full well you require at least python 3.2, I happily built pythonloader for (Debian jessie) python 2.7. Lo and behold, it actually works, except that when exiting, the process reliably segfaults.

The segfault happens in (again, Debian jessie) pystate.c:202, which is

        if (init)
            _PyThreadState_Init(tstate);

        HEAD_LOCK();
        tstate->next = interp->tstate_head; /* line 202 */
        interp->tstate_head = tstate;
        HEAD_UNLOCK();
    }

in the new_threadstate function -- so, I'd guess the list of threads is getting corrupted somewhere along the way (is threading absolutely necessary in pythonloader?).

The extension is trivial and does no imports, let alone threads:

def on_send_request(web_page, request, redirect):
    if request.get_uri().endswith(".css"):
        return True
    return False

def on_page_created(extension, web_page):
    web_page.connect("send-request", on_send_request)

def initialize(extension, arguments):
    extension.connect("page-created", on_page_created)

The versions are the ones from Debian jessie (in particular, gir1.2-webkit2-4-0 2.6.2).

So -- is this an expected failure for python2? If not, do you have a guess as to what might be wrong?

[aperezdc]

There is probably a bug somewhere. That being said, we do want the initialization that Python does when using threads, because the WebKitWebProcess is itself using threads. I am not 100% sure that it is needed, but it seems like the safer thing to do...

Honestly, I have no idea why this works in Python 3.x but not in 2.7.x —comparing the changes between both Pythons when it comes to the code that initializes and deinitializes threading support could shed some light. Unfortunately I have no time myself to investigate this anytime soon. Another option is trying to remove the calls to pyg_enable_threads() and PyEval_InitThreads() and see what happens.

[msdemlei]

I've investigated the thing just a tad further, still shy of installing debug versions of everything involved (if it exists).

So, the segfault happens when exit functions are run. I'm including the stacktrace as it is given I'm not using debug libraries. Incidentally, the stacktrace is invariant whether or not I call PyEval_InitThreads and pyg_enable_threads in pythonloader. I'd guess pygobject or something related will call them later anyway.

If first thought an explanation along the lines of http://stackoverflow.com/questions/31613626 could make sense: C-level atexit code runs when the interpreter has already been torn down. But that's actually not plausible, since obviously it's the interpreter itself that's calling __GI_exit.

But still, perhaps proposition is a fruitful idea that the trouble ensues when some exit code in libwebkit2gtk unreferences something (a python thread?) that doesn't have a valid python state any more? Given that the debug versoin of libwebkit2gtk-4.0 in jessie is a whopping gigabyte, I decided to see if perhaps that's already enough for someone actually familiar with the code to guess what's wrong. Is it?

#0  new_threadstate.lto_priv.3222 (interp=0x0, init=1)
    at ../Python/pystate.c:202
#1  0x080b111a in PyThreadState_New () at ../Python/pystate.c:213
#2  PyGILState_Ensure () at ../Python/pystate.c:600
#3  0xf67d9d07 in ?? () from /usr/lib/python2.7/dist-packages/gi/_gi.so
#4  0xf6b8d2b7 in g_closure_invalidate ()
   from /usr/lib/i386-linux-gnu/libgobject-2.0.so.0
#5  0xf6b8d05e in g_closure_unref ()
   from /usr/lib/i386-linux-gnu/libgobject-2.0.so.0
#6  0xf6ba6342 in g_signal_handlers_destroy ()
   from /usr/lib/i386-linux-gnu/libgobject-2.0.so.0
#7  0xf6b91a7d in ?? () from /usr/lib/i386-linux-gnu/libgobject-2.0.so.0
#8  0xf6b92349 in g_object_unref ()
   from /usr/lib/i386-linux-gnu/libgobject-2.0.so.0
#9  0xf3912974 in ?? () from /usr/lib/i386-linux-gnu/libwebkit2gtk-4.0.so.37
#10 0xf7543181 in __run_exit_handlers (status=status@entry=0, 
    listp=0xf76bb3c4 <__exit_funcs>, 
    run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#11 0xf75431dd in __GI_exit (status=0) at exit.c:104
#12 0x08177294 in Py_Exit (sts=0) at ../Python/pythonrun.c:1780
#13 0x081748e3 in handle_system_exit () at ../Python/pythonrun.c:1152

[PyErr_PrintEx is handling a SystemExit here]

#14 0x0817442a in PyErr_PrintEx () at ../Python/pythonrun.c:1162
#15 0x0806f61c in PyErr_Print () at ../Python/pythonrun.c:1065
#16 PyRun_SimpleFileExFlags (fp=0x841d498, 
    filename=0xffcd8ab1 "/usr/local/bin/pybrero", closeit=1, flags=0xffcd8228)
    at ../Python/pythonrun.c:953
#17 0x080db347 in Py_Main () at ../Modules/main.c:640
#18 0x080dadab in main (argc=2, argv=0xffcd8374) at ../Modules/python.c:23

[msdemlei]

So, here's a new piece of evidence that something is wrong. Not that it doesn't really tell me anything -- but perhaps someone else has an idea.

While everything else appears to be fine, now that I try to remove referer headers in a send-request handler like this:

request.get_http_headers().remove("referer")

things work "for a while" (~10 requests, say), and then I get a segfault in the rendering process in soup_message_headers_iter_next.

The main process then forks a new rendering process that, and that's what really scares me, reliably segfaults (i.e., the first request already fails the first time it reaches soup_message_headers_iter_next). How whatever state leads to that segfault gets, erm, inherited from one sibling process to the next might, I guess, give an idea what's going on.

Either way: If anyone has good advice on how to debug what's going on in the render process, I'll gladly take it. Bonus points if I can get around having more than one gig of debug symbols on my disk...

[msdemlei]

Well, I've moved on to python3, where there's not problem, and as python2 shouldn't be used for new projects anyway, I guess there's no real point in keeping this open.

[aperezdc]

@msdemlei: Sounds good to me. If someone else at some point needs Python 2.x, I will be happy to review patches but unfortunately I won't be having time myself to figure it out, nor I know enough of the Python/PyGObject internals as to be able to figure it out :disappointed: