Incomplete or inconsistent mod_wsgi configuration crashes Apache

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Install mod_wsgi as usual
2. Create mod_wsgi.conf in Apache configuration directory with just two lines:
LoadModule wsgi_module modules/mod_wsgi.so
WSGISocketPrefix run/wsgi

3. Reload Apache, it will start crashing

What is the expected output? What do you see instead?

Expected output: None

Observed: error_log becomes filled with messages:
[Mon Aug 31 15:16:15 2009] [notice] child pid 17740 exit signal
Segmentation fault (11)
[Mon Aug 31 15:16:15 2009] [notice] child pid 17741 exit signal
Segmentation fault (11)
[Mon Aug 31 15:16:15 2009] [notice] child pid 17742 exit signal
Segmentation fault (11)
[Mon Aug 31 15:16:15 2009] [notice] child pid 17743 exit signal
Segmentation fault (11)
[Mon Aug 31 15:16:15 2009] [notice] child pid 17744 exit signal
Segmentation fault (11)
[Mon Aug 31 15:16:15 2009] [notice] child pid 17745 exit signal
Segmentation fault (11)
[Mon Aug 31 15:16:15 2009] [notice] child pid 17746 exit signal
Segmentation fault (11)
...............

What version of the product are you using? On what operating system?

mod_wsgi 2.5
Apache 2.2.3
Python 2.4

OS: RedHat Enterprise Linux 5.3 (x86_64):
Linux psdev 2.6.18-128.7.1.el5 #1 SMP Wed Aug 19 04:00:49 EDT 2009 x86_64
x86_64 x86_64 GNU/Linux

Please provide any additional information below.

I also observe crashes in Apache when I forget to create wsgi script file
at the location that was specified in mod_wsgi.conf.

Original issue reported on code.google.com by andy.sal...@gmail.com on 1 Sep 2009 at 9:58

[GoogleCodeExporter]

This can occur where using mod_python in same Apache and mod_python using 
different Python version or 
either of mod_python of mod_wsgi using static Python library. Ensure you read 
through:

  http://code.google.com/p/modwsgi/wiki/InstallationIssues

Conflicts with shared libraries between want Python wants and what Apache 
and/or other Apache modules want, 
eg. PHP, can also cause crashes. Ensure you read through:

  http://code.google.com/p/modwsgi/wiki/ApplicationIssues

Ultimately the specific cause of such crashes can only be determined by getting 
a stack trace from gdb. You can 
follow procedure details at end of:

  http://code.google.com/p/modwsgi/wiki/DebuggingTechniques

It is likely one of these environment type issues rather than a problem in 
mod_wsgi itself. Would suggest triage 
of this issue be taken over to mod_wsgi mailing list on Google Groups as a 
result.

Original comment by Graham.Dumpleton@gmail.com on 2 Sep 2009 at 12:21

[GoogleCodeExporter]

Hi Graham,

thanks for response. I believe the there is something deeper happening here, 
not 
just shared library conflict. I played a bit more with and have some additional 
info for you:

- mod_python is not included in our configuration
- crashes only happen when I do things in this order:
  1. start apache without mod_wsgi.conf 
  2. add the above simple mod_wsgi.conf to configuration directory
  3. reload configuration using '/etc/init.d/httpd reload'
- if I use stop/start/restart instead of reload there are no crashes

I tried to catch the crash in a debugger, but the crash does not happen
when apache runs in single thread (-X option). I managed to produce core
files from multi-threaded apache and looked at them with debugger. All 
cores looks identical, here is the backtrace:

(gdb) thread apply all where

Thread 1 (process 2165):
#0  0x00002b350102e677 in sem_wait () from /lib64/libpthread.so.0
#1  0x00002b350a358148 in PyThread_acquire_lock () from 
/usr/lib64/libpython2.4.so.1.0
#2  0x00002b350a069378 in wsgi_python_child_init (p=0x0) at mod_wsgi.c:5456
#3  0x00002b34ff54597d in ap_run_child_init () from /usr/sbin/httpd
#4  0x00002b34ff5575ed in ?? () from /usr/sbin/httpd
#5  0x00002b34ff557b1a in ?? () from /usr/sbin/httpd
#6  0x00002b34ff55837d in ap_mpm_run () from /usr/sbin/httpd
#7  0x00002b34ff532e48 in main () from /usr/sbin/httpd

I also added "LogLevel info" to the configuration file but it did not add any 
useful info to the log files unfortunately.

Looking at the stack trace it seems that there are multithreading issues are at 
play here, even though there i sonly a single thread. And it's unclear why it's 
happening on reload only and not on restart. I have little knowledge about 
apache
internals to even try to guess.

Let me know if there is any other info that can be useful here.

Thanks,
Andy.

Original comment by andy.sal...@gmail.com on 2 Sep 2009 at 3:28

[GoogleCodeExporter]

What is the output from:

  httpd -l

  httpd -V

What other LoadModule lines do you have in Apache configuration files? Want 
this and above to know what MPM 
is used and what other Apache modules being loaded.

Original comment by Graham.Dumpleton@gmail.com on 2 Sep 2009 at 3:39

[GoogleCodeExporter]

Oh, can I also see the Apache startup message where it says in Apache error log 
what version of 
mod_wsgi/Python is being used.

Plus output from running:

  ldd mod_wsgi.so

and whether you have multiple Python installations on the system, and if 
multiple, where they are located.

Original comment by Graham.Dumpleton@gmail.com on 2 Sep 2009 at 3:41

[GoogleCodeExporter]

Here are all details:

-sh-3.2# httpd -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

-sh-3.2# httpd -V
Server version: Apache/2.2.3
Server built:   Jul  6 2009 05:29:28
Server's Module Magic Number: 20051115:3
Server loaded:  APR 1.2.7, APR-Util 1.2.7
Compiled using: APR 1.2.7, APR-Util 1.2.7
Architecture:   64-bit
Server MPM:     Prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=128
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="logs/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="logs/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

-sh-3.2# egrep -h '^ *LoadModule' /etc/httpd/conf/httpd.conf 
/etc/httpd/conf.d/*.conf
LoadModule auth_basic_module modules/mod_auth_basic.so
LoadModule auth_digest_module modules/mod_auth_digest.so
LoadModule authn_file_module modules/mod_authn_file.so
LoadModule authn_alias_module modules/mod_authn_alias.so
LoadModule authn_anon_module modules/mod_authn_anon.so
LoadModule authn_dbm_module modules/mod_authn_dbm.so
LoadModule authn_default_module modules/mod_authn_default.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule authz_user_module modules/mod_authz_user.so
LoadModule authz_owner_module modules/mod_authz_owner.so
LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
LoadModule authz_dbm_module modules/mod_authz_dbm.so
LoadModule authz_default_module modules/mod_authz_default.so
LoadModule ldap_module modules/mod_ldap.so
LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
LoadModule include_module modules/mod_include.so
LoadModule log_config_module modules/mod_log_config.so
LoadModule logio_module modules/mod_logio.so
LoadModule env_module modules/mod_env.so
LoadModule ext_filter_module modules/mod_ext_filter.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule expires_module modules/mod_expires.so
LoadModule deflate_module modules/mod_deflate.so
LoadModule headers_module modules/mod_headers.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule mime_module modules/mod_mime.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule info_module modules/mod_info.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule cache_module modules/mod_cache.so
LoadModule suexec_module modules/mod_suexec.so
LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule file_cache_module modules/mod_file_cache.so
LoadModule mem_cache_module modules/mod_mem_cache.so
LoadModule version_module modules/mod_version.so
LoadModule auth_kerb_module modules/mod_auth_kerb.so
LoadModule wsgi_module modules/mod_wsgi.so
LoadModule php5_module modules/libphp5.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

-sh-3.2# ldd /usr/lib64/httpd/modules/mod_wsgi.so 
        libpython2.4.so.1.0 => /usr/lib64/libpython2.4.so.1.0 (0x00002b6649232000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b6649564000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00002b664977f000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00002b6649984000)
        libm.so.6 => /lib64/libm.so.6 (0x00002b6649b87000)
        libc.so.6 => /lib64/libc.so.6 (0x00002b6649e0a000)
        /lib64/ld-linux-x86-64.so.2 (0x000000363fc00000)

I can't seem to find mod_wsgi version in Apache log, here is the only info that 
it
prints:

[Tue Sep 01 20:22:45 2009] [notice] core dump file size limit raised to
18446744073709551615 bytes
[Tue Sep 01 20:22:45 2009] [notice] suEXEC mechanism enabled (wrapper: 
/usr/sbin/suexec)
[Tue Sep 01 20:22:45 2009] [notice] Digest: generating secret for digest
authentication ...
[Tue Sep 01 20:22:45 2009] [notice] Digest: done
[Tue Sep 01 20:22:45 2009] [info] APR LDAP: Built with OpenLDAP LDAP SDK
[Tue Sep 01 20:22:45 2009] [info] LDAP: SSL support available
[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3433): Initializing Python.
[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3435): Attach interpreter ''.
[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3436): Attach interpreter ''.
[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3437): Attach interpreter ''.
[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3438): Attach interpreter ''.
[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3439): Attach interpreter ''.
[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3440): Attach interpreter ''.
[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3441): Attach interpreter ''.
[Tue Sep 01 20:22:45 2009] [notice] Apache configured -- resuming normal 
operations
[Tue Sep 01 20:22:45 2009] [info] Server built: Jul  6 2009 05:29:28
[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3442): Attach interpreter ''.

Anyway, mod_wsgi version is 2.5, I have built it myself from the sources.

We have only the standard RedHat version of Python in the system and this is 
version 2.4 installed in /usr (/usr/lib64/python2.4)

Hope this helps.

Andy.

Original comment by andy.sal...@gmail.com on 2 Sep 2009 at 4:13

[GoogleCodeExporter]

It looks like when I start apache with mod_wsgi.conf I can see all this 
messages in
log file:

[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3433): Initializing Python.
[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3435): Attach interpreter ''.
[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3436): Attach interpreter ''.
[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3437): Attach interpreter ''.
[Tue Sep 01 20:22:45 2009] [info] mod_wsgi (pid=3438): Attach interpreter ''.
....

But when I start apache without mod_wsgi.conf and then add it and reload none 
of those messages appear. Seems like Python is not getting initialized in this 
case. Just a guess of course.

Andy.

Original comment by andy.sal...@gmail.com on 2 Sep 2009 at 4:19

[GoogleCodeExporter]

Can you comment out the following LoadModule lines.

LoadModule logio_module modules/mod_logio.so

LoadModule cache_module modules/mod_cache.so

LoadModule disk_cache_module modules/mod_disk_cache.so
LoadModule file_cache_module modules/mod_file_cache.so
LoadModule mem_cache_module modules/mod_mem_cache.so

LoadModule php5_module modules/libphp5.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

Restart and see if problem persists. If problem goes away, put them back in a 
few at a time to narrow down 
which causes it.

Hopefully commenting them out will not require you to comment out other bits of 
Apache configuration.

You can use:

  apachectl -t

to test whether lack of module would cause configuration phase to die.

The startup line should look like:

[Mon Aug 31 12:14:42 2009] [notice] Apache/2.2.11 (Unix) mod_ssl/2.2.11 
OpenSSL/0.9.8k DAV/2 
mod_wsgi/3.0c5-TRUNK Python/2.6.1 configured -- resuming normal ope

Original comment by Graham.Dumpleton@gmail.com on 2 Sep 2009 at 4:21

[GoogleCodeExporter]

I commented out all modules that you mentioned but it did not change anything, 
the crashes are still there.

My log file does not show any message similar to what you have on startup.

Andy

Original comment by andy.sal...@gmail.com on 2 Sep 2009 at 6:11

[GoogleCodeExporter]

Could you try mod_wsgi-3.0c4.tar.gz from mod_wsgi download site.

After having done that, set:

  WSGIVerboseDebugging On
  LogLevel debug

at global server scope in Apache configuration file. Ie., outside of 
VirtualHost.

First see if this crashes or not, but secondly would like then to see all the 
Apache error log messages from time 
of start until when first segmentation faults start occurring.

Original comment by Graham.Dumpleton@gmail.com on 2 Sep 2009 at 6:34

[GoogleCodeExporter]

I have built 3.0c4 and retried all my tests with it. I do
not see any change in the crashing behavior, except that 
now it aborts with signal 6 (Aborted) instead of 11 (Segmentation 
Fault). I turned on debugging and I'm attaching error_log to this 
message. The log corresponds to the following events:

1. start apache without mod_wsgi.conf
2. created mod_wsgi.conf with this content:

LoadModule wsgi_module modules/mod_wsgi.so
WSGISocketPrefix run/wsgi
WSGIVerboseDebugging On

3. Reloaded apache with /etc/init.d/httpd reload'
4. Saw that error_log get flooded with error messages
5. Restarted apache with /etc/init.d/httpd restart' without removing 
mod_wsgi.conf

Let me know if there is anything else you want to try.

Andy

Original comment by andy.sal...@gmail.com on 2 Sep 2009 at 3:39

Attachments:

• error_log.gz

[GoogleCodeExporter]

It is failing because of this:

httpd: Python/pystate.c:493: PyGILState_Ensure: Assertion 
`autoInterpreterState' failed.

The code in Python related to this is:

        PyThreadState *tcur;
        /* Note that we do not auto-init Python here - apart from
           potential races with 2 threads auto-initializing, pep-311
           spells out other issues.  Embedders are expected to have
           called Py_Initialize() and usually PyEval_InitThreads().
        */
        assert(autoInterpreterState); /* Py_Initialize() hasn't been called! */

Thus, Python things that Py_Initialize() has never been called.

This can only mean one of three things.

The first is that module loading is somehow broken in Apache or due to some 
issue with your OS and the check:

    if (!Py_IsInitialized() || !initialized) {
        ...
        Py_Initialize();
    }

That is, it thinks Python is already initialised when it isn't.

I have seen this issue before in broken Apple supplied Apache 1.3 on MacOS X.

You could edit mod_wsgi.c and look for:

#if defined(DARWIN) && (AP_SERVER_MAJORVERSION_NUMBER < 2)
    static int initialized = 0;
#else
    static int initialized = 1;
#endif

Change the second initialisation to be '0' as well.

The second option is that Apache for some reason isn't calling the mod_wsgi 
wsgi_hook_init() function, which has a 
side effect of initialising Python. I would have expected lots of nasty things 
to happen if that wasn't the case and 
wouldn't even have got to trying to use Python later, as likely would have 
crashed before that point.

Lastly, there is some other Apache module loaded which is trigger a function 
which is in turn calling Python code 
before it is initialised. This is highly unlikely as all names in mod_wsgi and 
Python are prefix, so hard to inadvertantly 
do.

I would suggest changing the initialisation, but also add into that function 
after variable initialisation:

  fprintf(stderr, "wsgi_python_init\n");
  fflush(stderr);

Also add at start of wsgi_hook_init() function, after variable initialisation:

  fprintf(stderr, "wsgi_hook_init\n");
  fflush(stderr);

Note that there are two copies of that function, one for Apache 1.3 and another 
for Apache 2.X. Ensure you make 
change in correct one.

Then see if these messages come out in Apache error logs.

Alternatively, simply consider ensuring you are running up to date Apache and 
patch revision for that version of 
Python as your installations could be broken in some way.

Original comment by Graham.Dumpleton@gmail.com on 2 Sep 2009 at 11:44

[GoogleCodeExporter]

OK, I've done all exercises (with 3.0c4):

- changed initialized to 0 and added debugging output after it:

#if defined(DARWIN) && (AP_SERVER_MAJORVERSION_NUMBER < 2)
    static int initialized = 0;
#else
    static int initialized = 0;
#endif
  fprintf(stderr, "wsgi_python_init\n");
  fflush(stderr);

- added debugging output to wsgi_hook_init (for apache 2.0):

  fprintf(stderr, "wsgi_hook_init\n");
  fflush(stderr);

- the net result is it still crashes all the same

I'm attaching error_log again, it was generated again with the same sequence:
- start apache without mod_wsgi.conf
- add mod_wsgi.conf
- reload apache, see how it crashes
- restart apache, crashes disappear

What is interesting in the log file is that on 'restart' wsgi_hook_init is 
called
twice and on second call it also calls wsgi_python_init. On 'reload' 
wsgi_hook_init
is only called once and it does not call wsgi_python_init.

Original comment by andy.sal...@gmail.com on 3 Sep 2009 at 5:00

Attachments:

• error_log.gz

[GoogleCodeExporter]

That wsgi_hook_init called twice first time is normal. There is actually code 
in there to detect that and only 
trigger actions on second call, such as wsgi_python_init.

I will have to check whether wsgi_hook_init should be called twice on second 
time. I would expect it probably 
should but for some reason isn't and since Python initialisation only occurring 
on second call would, is not 
being done.

Thus, seems something wrong/odd with your Apache installation. Any chance you 
are able to upgrade 
Apache.

I'll look at the logs and see what else I can work out from it.

Original comment by Graham.Dumpleton@gmail.com on 3 Sep 2009 at 11:57

[GoogleCodeExporter]

Sorry for confusion, I probably explained it wrong. What I see is that when I do
'reload' wsgi_hook_init is called once only, and wsgi_python_init is not called 
at
all. On 'restart' wsgi_hook_init is called twice and second time it calls
wsgi_python_init as expected. This is easy to see in the log that I sent.

We have a stock Apache from RedHat RPMS and pretty fresh install, I doubt there 
is
anything wrong with it.

Original comment by andy.sal...@gmail.com on 4 Sep 2009 at 5:19

[GoogleCodeExporter]

The confusion is not your fault, but stupid Linux distributions. When you use 
'apachectl' directly, the options are 
'stop', 'start' and 'restart'. The init scripts that Linux uses have 'reload' 
match 'restart' of 'apachectl' (I think) and 
'reload' actually does a 'stop' and 'start' of 'apachectl'. At least I think 
that is how it goes, I can't remember right 
now. Maybe if you use 'apachectl' direct we can avoid the confusion of terms.

If I go back and add extra logging in wsgi_hook_init() so it says:

    fprintf(stderr, "wsgi_hook_init\n");
    fprintf(stderr, "Py_IsInitialized() = %d\n", Py_IsInitialized());
    fflush(stderr);

and in wsgi_python_init() so it says:

    fprintf(stderr, "wsgi_python_init\n");
    fprintf(stderr, "Py_IsInitialized() = %d\n", Py_IsInitialized());
    fprintf(stderr, "initialized = %d\n", initialized);
    fflush(stderr);

If I do an 'apachectl stop' and then fresh 'apachectl start', for the 'start' I 
see:

  wsgi_hook_init
  Py_IsInitialized() = 0
  wsgi_hook_init
  Py_IsInitialized() = 0
  wsgi_python_init
  Py_IsInitialized() = 0
  initialized = 1

If I know do a 'apachectl restart' I again see:

  wsgi_hook_init
  Py_IsInitialized() = 0
  wsgi_hook_init
  Py_IsInitialized() = 0
  wsgi_python_init
  Py_IsInitialized() = 0
  initialized = 1

So, wsgi_hook_init() should always be called twice. If that isn't occurring 
then there is something wrong with 
your Apache installation.

If the module init functions aren't being called twice like that, be quite 
surprised if some other Apache modules 
weren't working properly either.

Original comment by Graham.Dumpleton@gmail.com on 5 Sep 2009 at 10:49

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

As this is an isolated case, assuming that this was a problem with 
Apache/Python installation on system being 
used, perhaps related to mixing of 32bit/64bit compiled objects or libraries. 
Closing issue as can't see anything 
else that can be done and reporter didn't come back with any final outcome as 
to whether resolved or not.

Original comment by Graham.Dumpleton@gmail.com on 18 Nov 2009 at 6:38

• Changed state: WontFix
• Added labels: Type-Other
• Removed labels: Type-Defect

[GoogleCodeExporter]

May be similar to:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=558608

Still no clearer indication about what outside of mod_wsgi could be causing 
Apache not to initialise mod_wsgi 
properly, if that is indeed what is the cause.

Original comment by Graham.Dumpleton@gmail.com on 1 Dec 2009 at 10:48