jcc.initVM() doesn’t return when mod_wsgi is configured as daemon mode

[GoogleCodeExporter]

I am using mod-wsgi with django, and in django I use pylucene to do full
text search.

While mod-wsgi is configured to be embedded mode, there is no problem at all.
But when mod-wsgi is configured to be daemon mode, the apache just gets stuck, 
and the browser just keep loading but nothing appears.

Then I identity the problem to be the jcc.initVM().
Here is my wsgi script:

    import os, sys, jcc
    sys.stderr.write('jcc.initVM\n')
    jcc.initVM()
    sys.stderr.write('finished jcc.initVM\n')
    ....

After I restart my apache, and make a request from my browser, I find that
/var/log/apache2/error.log 
only has:

    jcc.initVM

Meaning that it gets stuck at the line jcc.initVM(). (If the mod_wsgi is
configured as embedded mode, there is no problem.)

And here is my /etc/apache2/sites-available/default:

    WSGIDaemonProcess site user=ross group=ross threads=1
    WSGIProcessGroup site
    WSGIScriptAlias / /home/ross/apache/django.wsgi

    <Directory /home/ross/apache/>
      Order deny,allow
      Allow from all
    </Directory>

And finally, I find out that in the source code of jcc (jcc.cpp), it hangs
at the function:

    JNI_CreateJavaVM(&vm, (void **) &vm_env, &vm_args)

How to solve the problem?

Program versions:

    libapache2-mod-wsgi 2.3-1
    jcc 2.1
    python 2.5
    Apache 2.2.9-8ubuntu3
    Ubuntu 8.10

Original issue reported on code.google.com by ph.tc...@gmail.com on 14 Feb 2009 at 4:43

[GoogleCodeExporter]

Probably same as:

  http://code.google.com/p/modwsgi/issues/detail?id=87

No simple solution yet except use a wrapper program around your program that 
unblocks signals.

Original comment by Graham.Dumpleton@gmail.com on 14 Feb 2009 at 5:01

[GoogleCodeExporter]

I am sorry that I still don't know how to work around.

django.wsgi:
import os
os.system('/home/ross/apache/run.py')

run.py:
#!/usr/bin/env python
import os, sys
import jcc
sys.stderr.write('jcc.initVM\n')
jcc.initVM()
sys.stderr.write('finished jcc.initVM\n')
sys.path.append('/home/ross/voofie/')
sys.path.append('/home/ross/')
os.environ['DJANGO_SETTINGS_MODULE'] = 'voofie.settings'
import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

It won't work. Could you give me some suggestion?

Original comment by ph.tc...@gmail.com on 14 Feb 2009 at 8:17

[GoogleCodeExporter]

It isn't that simple unfortunately. The wrapper program needs to be a C program 
as Python doesn't have access to 
required functions. Specifically, calls need to be made to either of the C 
functions sigblock() or sigprocmask() to 
restore ability of the thread that exec'd the program to receive signals.

I'll have another look at whether there might be a fiddle that can be done in 
mod_wsgi to allow it to work, but don't 
think there is. The code in mod_wsgi has to be written a completely different 
way and I just haven't got to that yet.

Original comment by Graham.Dumpleton@gmail.com on 14 Feb 2009 at 9:07

[GoogleCodeExporter]

Thank you very much. 

With reference to Brennan's exec.c, I write the following code using cython and
compiled that.

cpyrun.pyx:

cdef extern from "signal.h":
        ctypedef struct sigset_t:
                pass
        int sigfillset(sigset_t *set)
        int sigprocmask(int __how, sigset_t *set, sigset_t *oset)
        int SIG_UNBLOCK

import os, sys
import jcc
from django.core.handlers.wsgi import WSGIHandler
def run_wsgi():
        cdef sigset_t set
        sigfillset(&set)
        sigprocmask(SIG_UNBLOCK, &set, NULL)
        sys.stderr.write('jcc.initVM\n')
        jcc.initVM()
        sys.stderr.write('finished jcc.initVM\n')
        sys.path.append('/home/ross/site/')
        sys.path.append('/home/ross/')
        os.environ['DJANGO_SETTINGS_MODULE'] = 'site.settings'
        application = WSGIHandler()

And my django.wsgi:

import sys
sys.path.append('/home/ross/site/apache')
import cpyrun
cpyrun.run_wsgi()

However, I still get the same error, that the program still can't pass 
jcc.initVM.
Does that mean that I used the sigprocmask() wrongly? (I don't have much idea 
about
that, I just copy that from the exec.c and change that to cython)

Original comment by ph.tc...@gmail.com on 14 Feb 2009 at 10:39

[GoogleCodeExporter]

I'd still have:

django.wsgi:
import os
os.system('/home/ross/apache/run.py')

run.py:
#!/usr/bin/env python
...

and have signal unblocking in separate run.py program rather than in mod_wsgi 
daemon process as signals still going to 
likely be captured by Apache/mod_wsgi and not end up being handled by your 
program.

This is all assuming it is the same problem with SIGPWR.

Original comment by Graham.Dumpleton@gmail.com on 14 Feb 2009 at 10:57

[GoogleCodeExporter]

Sorry, I don't get what you mean in your comment 5.
In comment 3, you said I have to use sigblock() or sigprocmask() to unblock 
signals,
and I did that with comment 4's code. However it still didn't work out.
In comment 5, you referred back to the code in comment 2, which you already said
won't work since I didn't use sigblock() or sigprocmask().
Could you have a look to comment 4's code, and give me some feedback if the 
code use
sigprocmask() correctly? (Since I am not good in this.) 
Thank you very much.

Original comment by ph.tc...@gmail.com on 16 Feb 2009 at 2:25

[GoogleCodeExporter]

The point I was making was that you moved the unblocking of signals back into 
the WSGI script. Ie., you have WSGI script that runs inside of Apache 
process importing the module directly that tries to unblock signals. This 
possibly will not work. Your WSGI script file still has to use os.system() to 
execute a separate external script, with that external script containing the 
unblocking and then doing the Java stuff. I don't know how the C wrapping 
tool you are using works. But something to the effect of:

django.wsgi:
import os
os.system('/home/ross/apache/run.py')

run.py:
#!/usr/bin/env python
import sys
import jcc

import cpyrun
cdef sigset_t set
cpyrun.sigfillset(&set)
cpyrun.sigprocmask(SIG_UNBLOCK, &set, NULL)

sys.stderr.write('jcc.initVM\n')
jcc.initVM()
sys.stderr.write('finished jcc.initVM\n')
..... other stuff you need to do

Original comment by Graham.Dumpleton@gmail.com on 16 Feb 2009 at 2:55

[GoogleCodeExporter]

if I do that:
django.wsgi:
import os
os.system('/home/ross/apache/run.py')

It complains that:
[error] [client 127.0.0.1] mod_wsgi (pid=5687): Target WSGI script
'/home/ross/apache/django.wsgi' does notcontain WSGI application 'application'.

Original comment by ph.tc...@gmail.com on 16 Feb 2009 at 3:22

[GoogleCodeExporter]

I don't mean literally that. I only copied that from what you had said in 
comment 2. I took that only as indication of what you were going 
to have your Django application do, it may not even be in the WSGI script file 
but in some code associated with your function which is 
triggered by the URL when requested.

If it was in the WSGI script file, then you are perhaps going about this all 
the wrong way anyway. Remember that Apache/mod_wsgi can 
run as multiprocess configuration. The last thing you possibly want is this 
stuff from from inside of the web server processes anyway, 
especially if having multiple copies running is a bad thing.

Thus, you migher be better off having a standalone process that is executed 
distinct from the actual Django instance, that sets 
environment variables, imports Django infrastructure then does what it needs 
to. That is, it would actually serve requests.

Anyway, I am guess here as have no idea how pylucene works or what it really 
does. Since it mentions a JNI function I am just assuming it 
is Java stuff, which as per previous report doesn't work properly because of 
signals being blocked.

The question now is, does pylucence have to be in the web server processes. Can 
it just run in its own process with it perhaps populating 
database with index information which is then access via the web server 
processes?

Original comment by Graham.Dumpleton@gmail.com on 16 Feb 2009 at 3:40

[GoogleCodeExporter]

I believe I have a suitable fix for the underlying signal problem, presuming 
that is what is affecting you.

I had been holding off on this issue as felt I needed to first fix:

  http://code.google.com/p/modwsgi/issues/detail?id=102

which touches related code, so wanted to avoid having to make two sets of 
changes.

I'll look at doing the interim change for the signal problem first though even 
if have to change it again later.

Original comment by Graham.Dumpleton@gmail.com on 16 Feb 2009 at 9:29

[GoogleCodeExporter]

Thank you very much for your quick response and useful suggestion. I think I 
will
move the searching code to a totally new program let it communicate with the 
program
in mod_wsgi.

Original comment by ph.tc...@gmail.com on 17 Feb 2009 at 2:40

[GoogleCodeExporter]

One thing forgot to mention to try in case this is nothing to do with signals. 
Deletegate WSGI application to 
run in main Python interpreter.

  WSGIApplicationGroup %{GLOBAL}

Some third party extension modules for Python use simplified GIL API and will 
only work in main interpreter.

See:

  http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simplified_GIL_State_API

I have implemented fix for avoiding signal problem, but have to still validate 
it and commit it back to trunk.

Original comment by Graham.Dumpleton@gmail.com on 19 Feb 2009 at 10:20

[GoogleCodeExporter]

Marking this as duplicate, specifically of:

  http://code.google.com/p/modwsgi/issues/detail?id=87

until can be determined otherwise. The signal issue has now been addressed.

This issue may also have been connected with code simply not being run in main 
interpreter.

Original comment by Graham.Dumpleton@gmail.com on 20 Feb 2009 at 11:09

• Changed state: Duplicate