Apache Error log showing "[notice] child pid [pidnumber] exit signal Segmentation fault (11)"

[rcbarnett-zz]

MODSEC-87: A particular URL causes Apache to Segfault with the error shown in the summary.

It ocurrs during the response body processing phase so as a workaround we have added the following to Apache to disable this processing:

[rcbarnett-zz]

Original reporter: tyro

[rcbarnett-zz]

tyro: SecResponseBodyAccess Off

[rcbarnett-zz]

brectanus: Since I do not have the same binary as you I cannot get a good backtrace from those core dumps. Would you please provide a full backtrace?

Run: gdb /path/to/bin/httpd /path/to/core

In gdb do this and attach the output: thread apply all bt full

Or just do this and attach the bt.txt file:

gdb /path/to/bin/httpd /path/to/core > bt.txt <<EOT thread apply all bt full EOT

Also, do you know what the request was that caused this, or is it always for a particular URL? Can you duplicate it? If so please send details on duplicating (full request).

[rcbarnett-zz]

tyro: 3 different Apache core dumps attached from the same Segfault ocurring at different times.

[rcbarnett-zz]

brectanus: Also, please try 2.5.10-dev3 to see if this fixes the issue. I am holding up the release of 2.5.10 (was going to release today) for this issue. So, if you can respond quickly, please do (or comment that you cannot).

[rcbarnett-zz]

tyro: Thanks for your fast replies, sorry i could not get back to you in time. I'm afraid this is a production system and organising to gain the information you requested above will take some time to get setup on our test systems.

I can confirm that it was always a particular URL that was able to reproduce this Segfault.

Thank you.

[rcbarnett-zz]

brectanus: Thanks for letting me know it may be a while. Does this mean you can get this info within say a week? If so, I'll hold up 2.5.10, but if not, then I'll release it now. Unfortunately there is not much I can get out of the cores without your binaries. If you have the same binaries on a test system, then you can use the production cores with gdb on the test system to get a backtrace there.

Also, any more details on the request and your config causing this may also help (maybe I can then duplicate it here).

thanks again for the report and the prompt reply.

[rcbarnett-zz]

tyro: OK, i ran the following:

gdb /path/to/bin/httpd /path/to/core > bt.txt <<EOT thread apply all bt full EOT

and have attached the bt.txt file as well as the messages displayed when the command was run.

I hope that helps.

[rcbarnett-zz]

tyro: Hello,

1) I couldn't say exactly i'm afraid (less than half a meg i would say) 2) Dynamically generated text in tables 3) We would prefer to have blanket coverage of all pages 4) As follows: SecResponseBodyLimit 524288 SecRequestBodyInMemoryLimit 131072 SecResponseBodyLimitAction (there is no definition for this in our config) 5) I can see that rule would be more narrow than "SecResponseBodyAccess Off" which we currently have applied to the whole site, so we will keep that in mind. 6) I have attached your requested output from the following commands in bt-16-09-2009.txt: frame 3 print r print msr source /path/to/.gdbinit frame 3 dump_brigade bb_in dump_brigade msr->of_brigade dump_filters f dump_table msr->request_headers dump_table msr->response_headers dump_table msr->arguments dump_table msr->alerts

I hope that helps.

[rcbarnett-zz]

brectanus: The only thing that I see after a quick look is that the version of apr/apr-util that is loaded may be a different version than what mod_security was compiled against? Or maybe different than what Apache was built with?

Check this and make sure the compiled/loaded versions match:

/usr/sbin/httpd -V

Did you build ModSecurity yourself? Or is it a binary you installed? Perhaps you upgraded apr/apr-util versions after building ModSecurity and now they do not match (should be ok, but maybe there is an interaction)?

I'll take a closer look at the gdb output tomorrow.

Thanks.

[rcbarnett-zz]

brectanus: It is crashing when trying to write the response body to a buffer. Some questions for you:

1) How large (approx) is the response for this URL? 2) What type of data is returned from this URL? 3) Do you need to be inspecting the response for this URL? 4) What do you have the response body limits set to in ModSecurity (SecResponseBodyLimit, SecRequestBodyInMemoryLimit, SecResponseBodyLimitAction) ? 5) Does this rule help:

SecRule REQUEST_URI "^/the/uri/having/issues" \ "phase:1,nolog,pass,ctl:responseBodyAccess=off"

6) If you have the time, more data from the dump would be useful:

Run: gdb /usr/sbin/httpd /tmp/core.4035 And in gdb send the output of these commands:

frame 3 print r print msr

And if you have the .gdbinit file from the Apache source or grab .gdbinit from http://svn.apache.org/repos/asf/httpd/httpd/branches/2.2.x/.gdbinit, then these commands as well:

source /path/to/.gdbinit frame 3 dump_brigade bb_in

Thanks.

[rcbarnett-zz]

brectanus: Sorry, just saw that I left off some gdb commands in that last comment:

After "dump_brigade bb_in" also run these:

dump_brigade msr->of_brigade dump_filters f dump_table msr->request_headers dump_table msr->response_headers dump_table msr->arguments dump_table msr->alerts

Thanks.

[rcbarnett-zz]

tyro: Hi Brian,

Yes we built mod_security ourself, however we rebuilt it against the latest packages as part of our troubleshooting prior to creating this Jira. It is posssible that we did not rebuild it correctly, so if you see something, let me know. The output of the command is as follows.

[root@hostname user]# /usr/sbin/httpd -V Server version: Apache/2.2.3 Server built: Jul 15 2009 09:02:25 Server's Module Magic Number: 20051115:3 Server loaded: APR 1.2.7, APR-Util 1.2.7 Compiled using: APR 1.2.7, APR-Util 1.2.7 Architecture: 64-bit Server MPM: Prefork threaded: no forked: yes (variable process count) Server compiled with.... -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_SYSVSEM_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=128 -D HTTPD_ROOT="/etc/httpd" -D SUEXEC_BIN="/usr/sbin/suexec" -D DEFAULT_PIDLOG="logs/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_LOCKFILE="logs/accept.lock" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf"

[rcbarnett-zz]

brectanus: You still seeing this issue?

[rcbarnett-zz]

tyro: Hello,

We still have "SecResponseBodyAccess Off" set for the vhost. I have no reason to remove that again, as nothing else has changed.

[rcbarnett-zz]

tyro: Hello,

Just wondering if this has been resolved in any releases since?

[rcbarnett-zz]

bpinto: Hi Tyro,

Did u update your ModSec version ? Still seeing this seg fault ?

thanks

Breno

[rcbarnett-zz]

tyro: Hi Breno,

Thanks for checking. We have no plans to upgrade Production at the moment i'm sorry, so i'm unable to assist any further.

[rcbarnett-zz]

bpinto: Hi tyro,

Only looking for your debug info is difficult to know what is the problem. However seems to be a problem with apr lib, but i cannot say if it is a bug in the apr, or modsec bug. Please if you can provide the full url (maybe you can install some sniffer, live http plugin of firefox) that cause this problem i can try to reproduce it here.

thanks

Breno

[rcbarnett-zz]

tyro: Hi Breno,

Thank you for your efforts but we are unable to assist any further with this issue at this point. We can always refer back to this issue should we come back to it in the future.

Thank you.