mod_wsgi crashes apache on every request when loaded with -HUP

shangma / modwsgi

Automatically exported from code.google.com/p/modwsgi

0 stars 0 forks source link

What steps will reproduce the problem?

* install centos 5 or 6
* yum install httpd
* /etc/init.d/httpd start
* yum install mod_wsgi-3.2-3.el6 (centos6) or python26-mod_wsgi-3.3-1.el5 
(centos5)
* /etc/init.d/httpd reload

What is the expected output?

* Apache should continue to serve requests normally, with or without WSGI 
support (ideally WSGI would be loaded, but if that can't be done safely, then 
it's better not to load it).

What do you see instead?

* Every request to Apache hangs, and the child process terminates with a 
segmentation fault. Thus Apache is completely broken by loading the module 
after Apache initialisation.

This killed all our production servers when mod_wsgi.conf was installed as an 
unexpectedly wide side-effect of a Puppet change, and then this weekend log 
rotation caused Apache to HUP and freeze on all servers.

chris@lin-mail(~)$ sudo gdb `which httpd` 2174
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i686-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done.
Attaching to program: /usr/sbin/httpd, process 2174

warning: .dynamic section for "/lib/libz.so.1" is not at the expected address 
(wrong library or version mismatch?)
Reading symbols from /lib/i686/nosegneg/libm.so.6...(no debugging symbols 
found)...done.
Loaded symbols for /lib/i686/nosegneg/libm.so.6
Reading symbols from /lib/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib/libpcre.so.0
Reading symbols from /lib/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libselinux.so.1
Reading symbols from /usr/lib/libaprutil-1.so.0...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/libaprutil-1.so.0
Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libexpat.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libexpat.so.1
Reading symbols from /lib/libdb-4.7.so...(no debugging symbols found)...done.
Loaded symbols for /lib/libdb-4.7.so
Reading symbols from /usr/lib/libapr-1.so.0...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/libapr-1.so.0
Reading symbols from /lib/i686/nosegneg/libpthread.so.0...(no debugging symbols 
found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib/i686/nosegneg/libpthread.so.0
Reading symbols from /lib/i686/nosegneg/libc.so.6...(no debugging symbols 
found)...done.
Loaded symbols for /lib/i686/nosegneg/libc.so.6
Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libuuid.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /lib/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib/libfreebl3.so
Reading symbols from /lib/libnss_files.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_ldap.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib/libnss_ldap.so.2
Reading symbols from /lib/libldap-2.4.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib/libldap-2.4.so.2
Reading symbols from /lib/liblber-2.4.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib/liblber-2.4.so.2
Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libssl3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libssl3.so
Reading symbols from /usr/lib/libsmime3.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/libsmime3.so
Reading symbols from /usr/lib/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib/libnss3.so
Reading symbols from /usr/lib/libnssutil3.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/libnssutil3.so
Reading symbols from /lib/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /lib/libplds4.so
Reading symbols from /lib/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /lib/libplc4.so
Reading symbols from /lib/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /lib/libnspr4.so
Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /usr/lib/apr-util-1/apr_ldap-1.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/apr-util-1/apr_ldap-1.so
Reading symbols from /lib/i686/nosegneg/librt.so.1...(no debugging symbols 
found)...done.
Loaded symbols for /lib/i686/nosegneg/librt.so.1
Reading symbols from /etc/httpd/modules/mod_auth_basic.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_auth_basic.so
Reading symbols from /etc/httpd/modules/mod_auth_digest.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_auth_digest.so
Reading symbols from /etc/httpd/modules/mod_authn_file.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_authn_file.so
Reading symbols from /etc/httpd/modules/mod_authn_alias.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_authn_alias.so
Reading symbols from /etc/httpd/modules/mod_authn_anon.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_authn_anon.so
Reading symbols from /etc/httpd/modules/mod_authn_dbm.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_authn_dbm.so
Reading symbols from /etc/httpd/modules/mod_authn_default.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_authn_default.so
Reading symbols from /etc/httpd/modules/mod_authz_host.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_authz_host.so
Reading symbols from /etc/httpd/modules/mod_authz_user.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_authz_user.so
Reading symbols from /etc/httpd/modules/mod_authz_owner.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_authz_owner.so
Reading symbols from /etc/httpd/modules/mod_authz_groupfile.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_authz_groupfile.so
Reading symbols from /etc/httpd/modules/mod_authz_dbm.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_authz_dbm.so
Reading symbols from /etc/httpd/modules/mod_authz_default.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_authz_default.so
Reading symbols from /etc/httpd/modules/mod_ldap.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_ldap.so
Reading symbols from /etc/httpd/modules/mod_authnz_ldap.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_authnz_ldap.so
Reading symbols from /etc/httpd/modules/mod_include.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_include.so
Reading symbols from /etc/httpd/modules/mod_log_config.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_log_config.so
Reading symbols from /etc/httpd/modules/mod_logio.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_logio.so
Reading symbols from /etc/httpd/modules/mod_env.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_env.so
Reading symbols from /etc/httpd/modules/mod_ext_filter.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_ext_filter.so
Reading symbols from /etc/httpd/modules/mod_mime_magic.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_mime_magic.so
Reading symbols from /etc/httpd/modules/mod_expires.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_expires.so
Reading symbols from /etc/httpd/modules/mod_deflate.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_deflate.so
Reading symbols from /etc/httpd/modules/mod_headers.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_headers.so
Reading symbols from /etc/httpd/modules/mod_usertrack.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_usertrack.so
Reading symbols from /etc/httpd/modules/mod_setenvif.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_setenvif.so
Reading symbols from /etc/httpd/modules/mod_mime.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_mime.so
Reading symbols from /etc/httpd/modules/mod_dav.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_dav.so
Reading symbols from /etc/httpd/modules/mod_status.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_status.so
Reading symbols from /etc/httpd/modules/mod_autoindex.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_autoindex.so
Reading symbols from /etc/httpd/modules/mod_info.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_info.so
Reading symbols from /etc/httpd/modules/mod_dav_fs.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_dav_fs.so
Reading symbols from /etc/httpd/modules/mod_vhost_alias.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_vhost_alias.so
Reading symbols from /etc/httpd/modules/mod_negotiation.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_negotiation.so
Reading symbols from /etc/httpd/modules/mod_dir.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_dir.so
Reading symbols from /etc/httpd/modules/mod_actions.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_actions.so
Reading symbols from /etc/httpd/modules/mod_speling.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_speling.so
Reading symbols from /etc/httpd/modules/mod_userdir.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_userdir.so
Reading symbols from /etc/httpd/modules/mod_alias.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_alias.so
Reading symbols from /etc/httpd/modules/mod_substitute.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_substitute.so
Reading symbols from /etc/httpd/modules/mod_rewrite.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_rewrite.so
Reading symbols from /etc/httpd/modules/mod_proxy.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy.so
Reading symbols from /etc/httpd/modules/mod_proxy_balancer.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_balancer.so
Reading symbols from /etc/httpd/modules/mod_proxy_ftp.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_ftp.so
Reading symbols from /etc/httpd/modules/mod_proxy_http.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_http.so
Reading symbols from /etc/httpd/modules/mod_proxy_ajp.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_ajp.so
Reading symbols from /etc/httpd/modules/mod_proxy_connect.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_connect.so
Reading symbols from /etc/httpd/modules/mod_cache.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_cache.so
Reading symbols from /etc/httpd/modules/mod_suexec.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_suexec.so
Reading symbols from /etc/httpd/modules/mod_disk_cache.so...(no debugging 
symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_disk_cache.so
Reading symbols from /etc/httpd/modules/mod_cgi.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_cgi.so
Reading symbols from /etc/httpd/modules/mod_version.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_version.so
Reading symbols from /etc/httpd/modules/mod_ssl.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_ssl.so
Reading symbols from /usr/lib/libssl.so.10...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/libssl.so.10
Reading symbols from /usr/lib/libcrypto.so.10...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/libcrypto.so.10
Reading symbols from /lib/libgssapi_krb5.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib/libgssapi_krb5.so.2
Reading symbols from /lib/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib/libkrb5.so.3
Reading symbols from /lib/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /lib/libk5crypto.so.3...(no debugging symbols 
found)...done.
Loaded symbols for /lib/libk5crypto.so.3
Reading symbols from /lib/libkrb5support.so.0...(no debugging symbols 
found)...done.
Loaded symbols for /lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...(no debugging symbols 
found)...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /etc/httpd/modules/libphp5.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/libphp5.so
Reading symbols from /usr/lib/sse2/libgmp.so.3...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/sse2/libgmp.so.3
Reading symbols from /lib/libbz2.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libbz2.so.1
Reading symbols from /lib/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /usr/lib/libxml2.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /etc/httpd/modules/mod_wsgi.so...(no debugging symbols 
found)...done.
Loaded symbols for /etc/httpd/modules/mod_wsgi.so
Reading symbols from /usr/lib/libpython2.6.so.1.0...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/libpython2.6.so.1.0
Reading symbols from /lib/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib/libutil.so.1
Reading symbols from /usr/lib/php/modules/curl.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/php/modules/curl.so
Reading symbols from /usr/lib/libcurl.so.4...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/libcurl.so.4
Reading symbols from /lib/libidn.so.11...(no debugging symbols found)...done.
Loaded symbols for /lib/libidn.so.11
Reading symbols from /usr/lib/libssh2.so.1...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/libssh2.so.1
Reading symbols from /usr/lib/php/modules/fileinfo.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/php/modules/fileinfo.so
Reading symbols from /usr/lib/php/modules/json.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/php/modules/json.so
Reading symbols from /usr/lib/php/modules/mbstring.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/php/modules/mbstring.so
Reading symbols from /usr/lib/php/modules/phar.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/php/modules/phar.so
Reading symbols from /usr/lib/php/modules/zip.so...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib/php/modules/zip.so
0xf57fe416 in __kernel_vsyscall ()
Missing separate debuginfos, use: debuginfo-install 
httpd-2.2.15-15.el6.centos.1.i686
(gdb) set follow-fork child
(gdb) c
Continuing.
[New process 991]
[Thread debugging using libthread_db enabled]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb7226750 (LWP 991)]
0xb66cdc55 in ?? () from /etc/httpd/modules/mod_wsgi.so
(gdb) bt
#0  0xb66cdc55 in ?? () from /etc/httpd/modules/mod_wsgi.so
#1  0xb66ce3f0 in ?? () from /etc/httpd/modules/mod_wsgi.so
#2  0xb7720602 in ap_run_child_init ()
#3  0xb7736853 in ?? ()
#4  0xb7736e7e in ?? ()
#5  0xb7737e53 in ap_mpm_run ()
#6  0xb770aad2 in main ()

Original issue reported on code.google.com by ch...@aptivate.org on 14 Apr 2013 at 12:58

Impossible to tell. Most likely caused by a conflict with PHP wanting different shared library versions to what Python wants. PHP is well know for causing mod_wsgi to crash even though mod_wsgi works perfectly fine when PHP isn't loaded. Nothing can really do here as the specific cause isn't being identified.

Well, I can reproduce it easily, so what would you like me to try? Disabling PHP? mod_wsgi also works fine if it's loaded, along with PHP, when the server process starts. It just doesn't work if it's loaded later by a SIGHUP reload.

Here's a repro without mod_php loaded: chris@lin-mail(~)$ sudo gdb /usr/sbin/httpd 30040 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done. Attaching to program: /usr/sbin/httpd, process 30040 Reading symbols from /lib/i686/nosegneg/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/i686/nosegneg/libm.so.6 Reading symbols from /lib/libpcre.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpcre.so.0 Reading symbols from /lib/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libselinux.so.1 Reading symbols from /usr/lib/libaprutil-1.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libaprutil-1.so.0 Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/libexpat.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libexpat.so.1 Reading symbols from /lib/libdb-4.7.so...(no debugging symbols found)...done. Loaded symbols for /lib/libdb-4.7.so Reading symbols from /usr/lib/libapr-1.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libapr-1.so.0 Reading symbols from /lib/i686/nosegneg/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib/i686/nosegneg/libpthread.so.0 Reading symbols from /lib/i686/nosegneg/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/i686/nosegneg/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libuuid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libuuid.so.1 Reading symbols from /lib/libfreebl3.so...(no debugging symbols found)...done. Loaded symbols for /lib/libfreebl3.so Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /lib/libnss_ldap.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_ldap.so.2 Reading symbols from /lib/libldap-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libldap-2.4.so.2 Reading symbols from /lib/liblber-2.4.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/liblber-2.4.so.2 Reading symbols from /lib/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libresolv.so.2 Reading symbols from /usr/lib/libssl3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libssl3.so Reading symbols from /usr/lib/libsmime3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libsmime3.so Reading symbols from /usr/lib/libnss3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libnss3.so Reading symbols from /usr/lib/libnssutil3.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libnssutil3.so Reading symbols from /lib/libplds4.so...(no debugging symbols found)...done. Loaded symbols for /lib/libplds4.so Reading symbols from /lib/libplc4.so...(no debugging symbols found)...done. Loaded symbols for /lib/libplc4.so Reading symbols from /lib/libnspr4.so...(no debugging symbols found)...done. Loaded symbols for /lib/libnspr4.so Reading symbols from /usr/lib/libsasl2.so.2...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libsasl2.so.2 Reading symbols from /lib/libz.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libz.so.1 Reading symbols from /usr/lib/apr-util-1/apr_ldap-1.so...(no debugging symbols found)...done. Loaded symbols for /usr/lib/apr-util-1/apr_ldap-1.so Reading symbols from /etc/httpd/modules/mod_auth_basic.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_auth_basic.so Reading symbols from /etc/httpd/modules/mod_auth_digest.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_auth_digest.so Reading symbols from /etc/httpd/modules/mod_authn_file.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_authn_file.so Reading symbols from /etc/httpd/modules/mod_authn_alias.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_authn_alias.so Reading symbols from /etc/httpd/modules/mod_authn_anon.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_authn_anon.so Reading symbols from /etc/httpd/modules/mod_authn_dbm.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_authn_dbm.so Reading symbols from /etc/httpd/modules/mod_authn_default.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_authn_default.so Reading symbols from /etc/httpd/modules/mod_authz_host.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_authz_host.so Reading symbols from /etc/httpd/modules/mod_authz_user.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_authz_user.so Reading symbols from /etc/httpd/modules/mod_authz_owner.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_authz_owner.so Reading symbols from /etc/httpd/modules/mod_authz_groupfile.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_authz_groupfile.so Reading symbols from /etc/httpd/modules/mod_authz_dbm.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_authz_dbm.so Reading symbols from /etc/httpd/modules/mod_authz_default.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_authz_default.so Reading symbols from /etc/httpd/modules/mod_ldap.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_ldap.so Reading symbols from /etc/httpd/modules/mod_authnz_ldap.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_authnz_ldap.so Reading symbols from /etc/httpd/modules/mod_include.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_include.so Reading symbols from /etc/httpd/modules/mod_log_config.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_log_config.so Reading symbols from /etc/httpd/modules/mod_logio.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_logio.so Reading symbols from /etc/httpd/modules/mod_env.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_env.so Reading symbols from /etc/httpd/modules/mod_ext_filter.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_ext_filter.so Reading symbols from /etc/httpd/modules/mod_mime_magic.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_mime_magic.so Reading symbols from /etc/httpd/modules/mod_expires.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_expires.so Reading symbols from /etc/httpd/modules/mod_deflate.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_deflate.so Reading symbols from /etc/httpd/modules/mod_headers.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_headers.so Reading symbols from /etc/httpd/modules/mod_usertrack.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_usertrack.so Reading symbols from /etc/httpd/modules/mod_setenvif.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_setenvif.so Reading symbols from /etc/httpd/modules/mod_mime.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_mime.so Reading symbols from /etc/httpd/modules/mod_dav.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_dav.so Reading symbols from /etc/httpd/modules/mod_status.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_status.so Reading symbols from /etc/httpd/modules/mod_autoindex.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_autoindex.so Reading symbols from /etc/httpd/modules/mod_info.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_info.so Reading symbols from /etc/httpd/modules/mod_dav_fs.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_dav_fs.so Reading symbols from /etc/httpd/modules/mod_vhost_alias.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_vhost_alias.so Reading symbols from /etc/httpd/modules/mod_negotiation.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_negotiation.so Reading symbols from /etc/httpd/modules/mod_dir.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_dir.so Reading symbols from /etc/httpd/modules/mod_actions.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_actions.so Reading symbols from /etc/httpd/modules/mod_speling.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_speling.so Reading symbols from /etc/httpd/modules/mod_userdir.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_userdir.so Reading symbols from /etc/httpd/modules/mod_alias.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_alias.so Reading symbols from /etc/httpd/modules/mod_substitute.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_substitute.so Reading symbols from /etc/httpd/modules/mod_rewrite.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_rewrite.so Reading symbols from /etc/httpd/modules/mod_proxy.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_proxy.so Reading symbols from /etc/httpd/modules/mod_proxy_balancer.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_proxy_balancer.so Reading symbols from /etc/httpd/modules/mod_proxy_ftp.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_proxy_ftp.so Reading symbols from /etc/httpd/modules/mod_proxy_http.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_proxy_http.so Reading symbols from /etc/httpd/modules/mod_proxy_ajp.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_proxy_ajp.so Reading symbols from /etc/httpd/modules/mod_proxy_connect.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_proxy_connect.so Reading symbols from /etc/httpd/modules/mod_cache.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_cache.so Reading symbols from /etc/httpd/modules/mod_suexec.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_suexec.so Reading symbols from /etc/httpd/modules/mod_disk_cache.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_disk_cache.so Reading symbols from /etc/httpd/modules/mod_cgi.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_cgi.so Reading symbols from /etc/httpd/modules/mod_version.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_version.so Reading symbols from /etc/httpd/modules/mod_wsgi.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_wsgi.so Reading symbols from /usr/lib/libpython2.6.so.1.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libpython2.6.so.1.0 Reading symbols from /lib/libutil.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libutil.so.1 0xf57fe416 in __kernel_vsyscall () Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-15.el6.centos.1.i686 (gdb) set follow-fork-mode child (gdb) c Continuing. [New process 30640] [Thread debugging using libthread_db enabled] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb7232750 (LWP 30640)] 0xb6df6c55 in ?? () from /etc/httpd/modules/mod_wsgi.so (gdb) bt #0 0xb6df6c55 in ?? () from /etc/httpd/modules/mod_wsgi.so #1 0xb6df73f0 in ?? () from /etc/httpd/modules/mod_wsgi.so #2 0xb772c602 in ap_run_child_init () #3 0xb7742853 in ?? () #4 0xb7742e7e in ?? () #5 0xb7743e53 in ap_mpm_run () #6 0xb7716ad2 in main ()

And this is with all modules disabled, I think. chris@lin-mail(~)$ sudo gdb /usr/sbin/httpd 6435 GNU gdb (GDB) Red Hat Enterprise Linux (7.2-56.el6) Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i686-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done. Attaching to program: /usr/sbin/httpd, process 6435 Reading symbols from /lib/i686/nosegneg/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/i686/nosegneg/libm.so.6 Reading symbols from /lib/libpcre.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/libpcre.so.0 Reading symbols from /lib/libselinux.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libselinux.so.1 Reading symbols from /usr/lib/libaprutil-1.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libaprutil-1.so.0 Reading symbols from /lib/libcrypt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libcrypt.so.1 Reading symbols from /lib/libexpat.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libexpat.so.1 Reading symbols from /lib/libdb-4.7.so...(no debugging symbols found)...done. Loaded symbols for /lib/libdb-4.7.so Reading symbols from /usr/lib/libapr-1.so.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libapr-1.so.0 Reading symbols from /lib/i686/nosegneg/libpthread.so.0...(no debugging symbols found)...done. [Thread debugging using libthread_db enabled] Loaded symbols for /lib/i686/nosegneg/libpthread.so.0 Reading symbols from /lib/i686/nosegneg/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/i686/nosegneg/libc.so.6 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from /lib/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libdl.so.2 Reading symbols from /lib/libuuid.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libuuid.so.1 Reading symbols from /lib/libfreebl3.so...(no debugging symbols found)...done. Loaded symbols for /lib/libfreebl3.so Reading symbols from /lib/libnss_files.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_files.so.2 Reading symbols from /lib/libnss_ldap.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/libnss_ldap.so.2 Reading symbols from /etc/httpd/modules/mod_wsgi.so...(no debugging symbols found)...done. Loaded symbols for /etc/httpd/modules/mod_wsgi.so Reading symbols from /usr/lib/libpython2.6.so.1.0...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libpython2.6.so.1.0 Reading symbols from /lib/libutil.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libutil.so.1 0xf57fe416 in __kernel_vsyscall () Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-15.el6.centos.1.i686 (gdb) set follow-fork-mode child (gdb) c Continuing. [New process 6744] [Thread debugging using libthread_db enabled] Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0xb71e7750 (LWP 6744)] 0xb71acc55 in ?? () from /etc/httpd/modules/mod_wsgi.so (gdb) bt #0 0xb71acc55 in ?? () from /etc/httpd/modules/mod_wsgi.so #1 0xb71ad3f0 in ?? () from /etc/httpd/modules/mod_wsgi.so #2 0xb76e1602 in ap_run_child_init () #3 0xb76f7853 in ?? () #4 0xb76f7e7e in ?? () #5 0xb76f8e53 in ap_mpm_run () #6 0xb76cbad2 in main () (gdb)

shangma / modwsgi

mod_wsgi crashes apache on every request when loaded with -HUP #300