Segmentation fault due to expat library conflicts under mod_wsgi/3.3 and Python/2.5.2

GoogleCodeExporter commented 8 years ago

I built the latest version of openssl from source (I'm running debian 5.0 and 
the latest version under aptitude is several releases behind) and installed it 
in /usr/local/ssl.

I then rebuilt apache from source, using the --enable-ssl and 
--with-ssl=/usr/local/ssl options in configuration.

Next, I rebuilt mod wsgi using this configuration:

./configure --with-apxs=/usr/local/apache2/bin/apxs 
--with-python=/usr/bin/python

When I start apache, I get this notice, as expected:

[notice] Apache/2.2.17 (Unix) mod_ssl/2.2.17 OpenSSL/1.0.0a mod_wsgi/3.3 
Python/2.5.2 configured -- resuming normal operations

And when I request static pages from the server, under both http:// and 
https://, the appear as expected.

The moment I request a wsgi function, however, I get these errors:

[notice] child pid 28615 exit signal Segmentation fault (11)

The only thing new about my setup is the new version of openssl, which was 
built from source rather than using aptitude.

I also tried removing the older version of openssl with aptitude, and I 
re-built mod wsgi, but there was nothing explicit in its configuration that 
refers to openssl.

Is there anything else I can try?

Original issue reported on code.google.com by denis.pa...@gmail.com on 14 Nov 2010 at 10:23

GoogleCodeExporter commented 8 years ago

Is this for a hello world WSGI application or your application, using some 
large web framework?

If the latter, is it using any SSL modules directly or indirectly in Python 
code?

There may be an issue with Python modules using/expecting different SSL version.

Try using a WSGI hello world application if you aren't already:

  http://code.google.com/p/modwsgi/wiki/QuickConfigurationGuide#WSGI_Application_Script_File

Try working out exactly where it is crashing. See:

  http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With_GDB

Also, indicate how you have set up mod_wsgi, ie., embedded mode or daemon mode. 
Add the Apache configuration file snippets to issue.

Original comment by Graham.Dumpleton@gmail.com on 14 Nov 2010 at 10:35

GoogleCodeExporter commented 8 years ago

Hi, and thanks for your quick reply.

> Is this for a hello world WSGI application or your application, using some 
large web framework?

It's the former, i.e., all the wsgi scripts follow the pattern of the WSGI 
hello world application.

The httpd.conf file contains these lines:

LoadModule wsgi_module modules/mod_wsgi.so

WSGIPythonPath /usr/local/apache2/wsgi
WSGIScriptAlias /Contact /usr/local/apache2/wsgi/Contact.py
...
(etc.)
...

<Directory /usr/local/apache2/wsgi>
Order allow,deny
Allow from all
</Directory>

> Try working out exactly where it is crashing. See:
>   
http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With
_GDB

Thanks for this advice; I will try it now.

Original comment by denis.pa...@gmail.com on 14 Nov 2010 at 10:50

GoogleCodeExporter commented 8 years ago

The gdb backtrace looks like this:

(gdb) backtrace
#0  0x08135988 in EVP_PKEY_CTX_dup ()
#1  0x00000006 in ?? ()
#2  0xb78f97f0 in ?? ()
#3  0xbfa063d8 in ?? ()
#4  0xb7fb77b4 in ?? () from /usr/lib/python2.5/lib-dynload/_hashlib.so
#5  0xb7af8e8c in ?? ()
#6  0xb7fb77b4 in ?? () from /usr/lib/python2.5/lib-dynload/_hashlib.so
#7  0xb7af8e8c in ?? ()
#8  0x0812b142 in EVP_MD_CTX_copy_ex ()
#9  0xb7c8e265 in _PyObject_New () from /usr/lib/libpython2.5.so.1.0
#10 0xb7fb666d in ?? () from /usr/lib/python2.5/lib-dynload/_hashlib.so
#11 0xb7fb77b4 in ?? () from /usr/lib/python2.5/lib-dynload/_hashlib.so
#12 0x00000000 in ?? ()

From what I can tell, it seems to be a conflict between the version of openssl 
that python expects (old) and the version I used to build mod wsgi (latest).

I tried removing both openssl and libssl-dev with aptitude, then re-building 
openssl with the --prefix=/usr option, hoping that I could fool python into 
using the version of openssl I want, but the problem remains.

So I think my best course of action would be to get off debian and rebuild the 
image under ubuntu instead.

Thank you for the debugging suggestions.

Original comment by denis.pa...@gmail.com on 15 Nov 2010 at 5:53

GoogleCodeExporter commented 8 years ago

You likely need to rebuild Python from source code to use same SSL libraries 
that Apache was built against. Also ensure that Apache is using your SSL and 
not using its internal version.

Original comment by Graham.Dumpleton@gmail.com on 16 Nov 2010 at 3:20

GoogleCodeExporter commented 8 years ago

Actually, I now think the problem is related to the expat library, as described 
here: http://code.google.com/p/modwsgi/wiki/IssuesWithExpatLibrary

In my case, the apache version is:

# lsof -p 17431 | grep expat
httpd   17431 root  mem    REG  202,0  381845 352419 
/usr/local/apache2/lib/libexpat.so.0.5.0
# strings /usr/local/apache2/lib/libexpat.so.0 | grep expat_
expat_1.95.7

Whereas the python version is:

# python
Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyexpat
>>> pyexpat.version_info
(2, 0, 0)

Unfortunately, the prescribed solution using LD_PRELOAD didn't work:

# LD_PRELOAD=/usr/local/apache2/lib/libexpat.so.0.5.0 python
Python 2.5.2 (r252:60911, Jan 24 2010, 14:53:14) 
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyexpat
>>> pyexpat.version_info
(2, 0, 0)

Original comment by denis.pa...@gmail.com on 16 Nov 2010 at 11:55

GoogleCodeExporter commented 8 years ago

The expat issue should not affect Python 2.5 or later unless you are using 
version modified by Linux distribution and they broke the namespace prefixing 
now do to avoid symbol conflicts.

In other words, Python 2.5 and later protects against the problem.

Original comment by Graham.Dumpleton@gmail.com on 17 Nov 2010 at 12:05

GoogleCodeExporter commented 8 years ago

Well, the python I'm using is the default which came with the base system, and 
other than adding related packages through aptitude (e.g., python-all-dev, 
python-lxml, etc.), I haven't messed with it.

I also recreated the same setup under Fedora 13 and got the same error.

Fedora 13 has OpenSSL v. 1.0.0a included in the base system, so I wanted to 
eliminate it as a variable.

I was able to build Apache 2.2.17 and mod_wsgi 3.3 from source, and the simple 
"Hello World" script worked.

When I ran the "with expat" version, i.e., by deliberately importing pypexpat, 
I got this error in apache's log:

httpd: Objects/stringobject.c:115: PyString_FromString: Assertion `str != 
((void *)0)' failed.

A search brought me to the expat article on this wiki, and when I checked the 
Fedora environment, I saw a similar mismatch:

# lsof -p 18714 | grep expat
httpd   18714 root  mem    REG  202,0   362608 139352 
/usr/local/apache2/lib/libexpat.so.0.5.0
# strings /usr/local/apache2/lib/libexpat.so.0.5.0 | grep expat_
expat_1.95.7

# python
Python 2.6.4 (r264:75706, Jun  4 2010, 18:20:16) 
[GCC 4.4.4 20100503 (Red Hat 4.4.4-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyexpat
>>> pyexpat.version_info
(2, 0, 1)

Even more telling, when I tried LD_PRELOAD with python in Fedora, I got this 
error after trying to import pyexpat, which matched the apache error log:

# LD_PRELOAD=/usr/local/apache2/lib/libexpat.so.0.5.0 python
Python 2.6.4 (r264:75706, Jun  4 2010, 18:20:16) 
[GCC 4.4.4 20100503 (Red Hat 4.4.4-2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyexpat
python: Objects/stringobject.c:115: PyString_FromString: Assertion `str != 
((void *)0)' failed.
Aborted

Original comment by denis.pa...@gmail.com on 17 Nov 2010 at 12:22

GoogleCodeExporter commented 8 years ago

Okay, forgot that they don't need to have break namespace. Can also occur when 
Python has been built so as to use standalone expat shared library and not 
compile in the expat code shipped with Python.

I was aware of this as Apple do this on MacOS X 10.6 and some Linux 
distributions may do it also.

Do:

  ldd lib-dynload/pyexpat.so
  nm lib-dynload/pyexpat.so | grep XML_ExpatVersion

on the pyexpat.so in your Python installation.

If ldd shows dependency on libexpat.so and that symbol isn't name space prefix 
with Py??? and is specified as 'U' linkage, ie., undefined, then not local 
symbol and clashes can occur if Apache loads different expat version.

Apache would need to be rebuilt to use same version of distinct expat shared 
library.

I'll need to update the documentation to also warn about that.

Original comment by Graham.Dumpleton@gmail.com on 17 Nov 2010 at 12:34

Changed title: Segmentation fault due to expat library conflicts under mod_wsgi/3.3 and Python/2.5.2
Changed state: Accepted
Added labels: Component-Docs

GoogleCodeExporter commented 8 years ago

Original comment by Graham.Dumpleton@gmail.com on 17 Nov 2010 at 12:34

Added labels: Type-Other
Removed labels: Type-Defect

GoogleCodeExporter commented 8 years ago

Closing as the task to document libexpat linking issue already covered by issue 
#203.

Original comment by Graham.Dumpleton@gmail.com on 19 Mar 2012 at 10:45

Changed state: Duplicate

ramkrishanbhatt / modwsgi

Segmentation fault due to expat library conflicts under mod_wsgi/3.3 and Python/2.5.2 #213