Segmentation fault in ModSecurity 2.9 for Apache

[vizovitin]

Describe the bug

Segmentation fault in ModSecurity 2.9.7 for Apache on very specific POST requests.

Logs and dumps

I cannot provide full core dump or request parameters due to potential inclusion of information from a production server. However, here's the stack trace:

(gdb) where
#0  0x00007f33133b507f in modsecurity_request_body_end_raw (msr=0x7f33030f7ba8, error_msg=0x7ffc207520e8)
    at apache2/msc_reqbody.c:572
#1  0x00007f33133aa748 in modsecurity_request_body_end_urlencoded (error_msg=0x7ffc207520e8, msr=0x7f33030f7ba8)
    at apache2/msc_reqbody.c:622
#2  modsecurity_request_body_end (error_msg=0x7ffc207520e8, msr=0x7f33030f7ba8)
    at apache2/msc_reqbody.c:730
#3  read_request_body (error_msg=0x7ffc207520e8, msr=<optimized out>)
    at apache2/apache2_io.c:350
#4  hook_request_late (r=0x7f33114960a0)
    at apache2/mod_security2.c:1026
#5  0x0000564a3d81b9a8 in ap_run_fixups ()
#6  0x0000564a3d82e553 in ap_process_request_internal ()
#7  0x0000564a3d84f8e5 in ap_process_async_request ()
#8  0x0000564a3d84fb33 in ap_process_request ()
#9  0x0000564a3d84fe77 in ?? ()
#10 0x0000564a3d83be88 in ap_run_process_connection ()
#11 0x00007f3313eed128 in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#12 0x00007f3313eed4a1 in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#13 0x00007f3313eedda7 in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#14 0x0000564a3d8070e8 in ap_run_mpm ()
#15 0x0000564a3d806609 in main ()

(gdb) bt full
#0  0x00007f33133b507f in modsecurity_request_body_end_raw (msr=0x7f33030f7ba8, error_msg=0x7ffc207520e8)
    at apache2/msc_reqbody.c:572
        chunks = <optimized out>
        one_chunk = <optimized out>
        d = 0x564a3f7dc390 "\340s}\027\003\003\b@alswitch\",\"from\":\"meta\",\"options\":{\"H\":\"DISABLE\",\"S\":\"ENABLE\"},\"default\":\"H\",\"children\":[\"vibe_partial_units\"],\"hide_nodes\":[\"vibe_partial_units\"],\"value\":\"H\"},{\"label\":\"Select items for parti"...
        i = <optimized out>
        sofar = 0
#1  0x00007f33133aa748 in modsecurity_request_body_end_urlencoded (error_msg=0x7ffc207520e8, msr=0x7f33030f7ba8)
    at apache2/msc_reqbody.c:622
        invalid_count = 0
        invalid_count = <optimized out>
#2  modsecurity_request_body_end (error_msg=0x7ffc207520e8, msr=0x7f33030f7ba8)
    at apache2/msc_reqbody.c:730
        my_error_msg = 0x0
        metadata = <optimized out>
...
(gdb) print msr->msc_reqbody_length
$5 = 128070
(gdb) print msr->msc_reqbody_buffer
$4 = 0x564a3f7dc390 "\340s}\027\003\003\b@alswitch\",\"from\":\"meta\",\"options\":{\"H\":\"DISABLE\",\"S\":\"ENABLE\"},\"default\":\"H\",\"children\":[\"vibe_partial_units\"],\"hide_nodes\":[\"vibe_partial_units\"],\"value\":\"H\"},{\"label\":\"Select items for parti"...
(gdb) print i
$1 = <optimized out>
(gdb) print msr->msc_reqbody_chunks
$2 = (apr_array_header_t *) 0x0
(gdb) print sofar
$6 = 0

 543 static apr_status_t modsecurity_request_body_end_raw(modsec_rec *msr, char **error_msg) {
 ...
 567     /* Copy the data we keep in chunks into the new buffer. */
 568
 569     sofar = 0;
 570     d = msr->msc_reqbody_buffer;
 571     chunks = (msc_data_chunk **)msr->msc_reqbody_chunks->elts;
 572     for(i = 0; i < msr->msc_reqbody_chunks->nelts; i++) {
 573         if (sofar + chunks[i]->length <= msr->msc_reqbody_length) {
 574             memcpy(d, chunks[i]->data, chunks[i]->length);
 575             d += chunks[i]->length;
 576             sofar += chunks[i]->length;
 577         } else {
 578             *error_msg = apr_psprintf(msr->mp, "Internal error, request body buffer overflow.");
 579             return -1;
 580         }
 581     }

To Reproduce

N/A

Reproduced only on production server with very specific steps in a very customized WordPress. ModSecurity for Apache, blocking, with Comodo (free) ruleset, in Fast mode, with 210710 and 222212 security rule IDs switched off.

Actual behavior

Request fails with 502 Bad Gateway served by nginx (reverse proxy before Apache). In the backend the following is logged:

==> /var/www/vhosts/system/example.net/logs/proxy_error_log <==
2024/11/13 08:59:28 [error] 3413043#0: *178721 upstream prematurely closed connection while reading response header from upstream, client: 123.45.123.45, server: example.net, request: "POST /wp-json/wplms/v2/saveDraft/?post HTTP/2.0", upstream: "https://127.0.0.1:7081/wp-json/wplms/v2/saveDraft/?post", host: "example.net", referrer: "https://example.net/"

==> /var/log/apache2/error.log <==
[Wed Nov 13 08:59:29.122617 2024] [core:notice] [pid 1141769] AH00051: child pid 3437845 exit signal Segmentation fault (11), possible coredump in /etc/apache2

Expected behavior

Apache children should not crash due to ModSecurity.

Server (please complete the following information):

• ModSecurity version (and connector): 2.9.7
• WebServer: apache2 2.4.52-1ubuntu4.12 (OS vendor packages)
• OS (and distro): Ubuntu 22.04 x86_64 (likely ANY Linux)

Rule Set (please complete the following information):

• Running any public or commercial rule set? Comodo (free)
• What is the version number? 1.241

Additional context

Unfortunately, I cannot test the issue on 2.9.8 (as it is production server I don't own).

The following patch alleviates the issue, although it is most certainly "incorrect":

--- apache2/msc_reqbody.c.orig  2024-11-13 14:52:48.728063792 +0000
+++ apache2/msc_reqbody.c       2024-11-13 14:56:58.774675155 +0000
@@ -547,6 +547,11 @@

     *error_msg = NULL;

+    if (msr->msc_reqbody_chunks == NULL) {
+       *error_msg = apr_psprintf(msr->mp, "Internal error, request body chunks are NULL.");
+       return -1;
+    }
+
     /* Allocate a buffer large enough to hold the request body. */

     if (msr->msc_reqbody_length + 1 == 0) {

[airween]

Hi @vizovitin,

thanks for provided info. We try to investigate the root cause soon, but probably the given patch would be enough at first.

@marcstern what do you think?

[vizovitin]

JFYI: There might be a few other places in the same file with similar issue, but since I'm not acquainted with the code and they don't seem to trigger any issues, I chose to keep the patch to a minimum (i.e. patch only one usage).

[marcstern]

I wonder if the cause is not that modsecurity_request_body_end_raw() is called with a body that's not chunked-encoded. Why is it called inside modsecurity_request_body_end_urlencoded(), thus when body is url-encoded, and not when multipart-encoding is used? Shouldn't we solve the problem at a higher level (and add assert() all over the place)?

@vizovitin: can you show us an example of problematic request?

[vizovitin]

Unfortunately, I don't think I can. It is from a production server I don't control. The request itself is rather large and includes various authentication and other potentially private data.

If you need to check something specific I may still be able to check it.

[marcstern]

Can we have the request headers (you can remove any specific one, like cookies)?

[vizovitin]

Sure. I've redacted some stuff (mostly marked as REDACTED). Also POST data is a lot larger. I've also changed the domain address.

curl 'https://abcde.example.net/wp-json/wplms/v2/create/13377?post' \
  -H 'accept: */*' \
  -H 'accept-language: en-US,en;q=0.9' \
  -H 'content-type: text/plain;charset=UTF-8' \
  -H 'cookie: wordpress_test_cookie=WP%20Cookie%20check; wp_lang=en_US; other_cookies=REDACTED' \
  -H 'origin: https://abcde.example.net' \
  -H 'priority: u=1, i' \
  -H 'referer: https://abcde.example.net/' \
  -H 'sec-ch-ua: "Not/A)Brand";v="8", "Chromium";v="123", "Google Chrome";v="123"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "Windows"' \
  -H 'sec-fetch-dest: empty' \
  -H 'sec-fetch-mode: cors' \
  -H 'sec-fetch-site: same-origin' \
  -H 'user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/512.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/512.36' \
  --data-raw '{"course":"REDACTED","object":["REDACTED"],"token":"REDACTED","course_id":13377}'

[marcstern]

Can we have the headers "Content-Type" and "Transfer-Encoding" from the original request?

[vizovitin]

Content-Type is listed above. There was no Transfer-Encoding, apparently.

[marcstern]

Sorry, I meant Content-Length. As there was no Transfer-Encoding, I assume Content-Length was present. Can you check with curl -v? Also, what are the values of SecRequestBodyLimitAction, SecRequestBodyLimit, SecRequestBodyNoFilesLimit in your config? Is the body bigger than SecRequestBodyLimit/SecRequestBodyNoFilesLimit?

[vizovitin]

I just copied the request data from a browser developer console. So if the header is not there - I suppose it wasn't present.

There are no specified directives in the configuration:

# egrep -R '^\s*SecRequestBody' /etc/apache2/
/etc/apache2/plesk.conf.d/server.conf:  SecRequestBodyAccess Off
/etc/apache2/plesk.conf.d/server.conf.bak:      SecRequestBodyAccess Off
/etc/apache2/conf-available/modsec2.imunify.conf:  SecRequestBodyAccess On
/etc/apache2/conf-enabled/zz999_modsec2.imunify.conf:  SecRequestBodyAccess On

Can you check with curl -v?

content-length: 70888

You can also see above in the issue, which might be useful:

(gdb) print msr->msc_reqbody_length
$5 = 128070