zimmerle
commented
6 years ago Confirmed that this issue depending on have Nginx configured with a set of 3rd party modules. Not able to reproduce the issue in a plain version of ModSecurity.
Closed zimmerle closed 5 years ago
zimmerle
commented
6 years ago Confirmed that this issue depending on have Nginx configured with a set of 3rd party modules. Not able to reproduce the issue in a plain version of ModSecurity.
ghost
commented
6 years ago Thanks @zimmerle . I guess at this stage I will just unpick the build script of my nginx container until I find which specific flag / module / patch is causing it.
ghost
commented
6 years ago @zimmerle Still crashing. I have simplified the setup to help you reproduce and also removed all 3rd party modules and patches from the build script. Nginx is now built like this:
./configure \
--prefix=/usr/share/nginx \
--conf-path=/etc/nginx/nginx.conf \
--modules-path=/etc/nginx/modules \
--http-log-path=/var/log/nginx/access.log \
--error-log-path=/var/log/nginx/error.log \
--lock-path=/var/lock/nginx.lock \
--pid-path=/run/nginx.pid \
--http-client-body-temp-path=/var/lib/nginx/body \
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi \
--http-proxy-temp-path=/var/lib/nginx/proxy \
--http-scgi-temp-path=/var/lib/nginx/scgi \
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi \
--with-http_ssl_module \
--without-mail_pop3_module \
--without-mail_smtp_module \
--without-mail_imap_module \
--without-http_uwsgi_module \
--without-http_scgi_module \
--add-dynamic-module=$BUILD_PATH/ModSecurity-nginx-$MODSECURITY \
&& make || exit 1 \
&& make install || exit 1Nginx config file is now:
load_module /etc/nginx/modules/ngx_http_modsecurity_module.so;
daemon off;
worker_processes 2;
pid /run/nginx.pid;
worker_rlimit_nofile 523264;
events {
worker_connections 16384;
}
http {
sendfile on;
keepalive_timeout 75s;
include /etc/nginx/mime.types;
default_type text/html;
error_log /var/log/nginx/error.log notice;
server {
server_name _;
listen 80 default_server;
location / {
set $dummy 0;
if ($dummy) {
return 301 http://google.com;
}
modsecurity on;
modsecurity_rules_file /etc/nginx/modsecurity/modsecurity.conf;
proxy_pass http://google.com:80/;
}
}
}Reminder, the modsecurity_rules_file must be appended with a SecRuleUpdateTargetById
If I remove the if block it also stops crashing.
zimmerle
commented
6 years ago Hi @slabber,
I am re-opening the issue. I appreciate the detailed description, unfortunately, due to reasons bigger then my desire, I cannot work on that issue at this moment.
zimmerle
commented
5 years ago As of ModSecurity's commit: fa5f3784f24442ef64114aa57090fa5f1395a2e4 this is no longer an issue.
@slabber commented on Thu Oct 12 2017
I updated a rule by appending to the config:
SecRuleUpdateTargetById 942421 "!REQUEST_COOKIES:/example/"When I hit this rule the worker process dies with signal 11:
This is with core rule set 3.0.2, libmodsec 3.0.0rc1 and nginx compiled with:
nginx version: nginx/1.13.6 built by gcc 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.5) built with OpenSSL 1.0.2g 1 Mar 2016 TLS SNI support enabled configure arguments: --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/ngi nx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-tem p-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-htt p_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gzip_static_module --with-http_sub_module --with-http_v2_module --with-stream -- with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-file-aio --without-mail_pop3_module --without-mail_smtp_module --without-mail_imap_module --without-http_uwsgi_m odule --without-http_scgi_module --with-cc-opt='-g -O3 -flto -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 --param=ssp-buffer-size=4 -DTCP_F ASTOPEN=23 -Wno-error=strict-aliasing -fPIC -m64 -mtune=generic' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -fPIC -pie -Wl,-z,relro -Wl,-z,now' --add-module=/tmp/build/ngx_devel_kit-0.3.0 --add-module=/tmp/build/set-misc-nginx-module-0.31 --add-module=/tmp/build/nginx-module-vts-0.1.15 --add-module=/tmp/build/headers-more-nginx-module-0.32 --add-module=/tmp/build/nginx-goodi es-nginx-sticky-module-ng-08a395c66e42 --add-module=/tmp/build/nginx-http-auth-digest-7955af9c77598c697ac292811914ce1e2b3b824c --add-module=/tmp/build/ngx_http_substitutions_filter_module-bc 58cb11844bc42735bbaef7085ea86ace46d05b --add-module=/tmp/build/nginx-opentracing-fcc2e822c6dfc7d1f432c16b07dee9437c24236a --add-dynamic-module=/tmp/build/ModSecurity-nginx-a2a5858d249222938c 2f5e48087a922c63d7f9d8@slabber commented on Thu Oct 12 2017
modsecurity.conf.txt
@zimmerle commented on Fri Oct 20 2017
Hi @slabber
Thanks for the report.
On 93e18ca5ea18659af58a9f23d057917a01b9867d and 30797a458b1f8e42905538219ea75ddc9c719a76 we have changed the way that the parser understands the element selection while using a regex. That changes may affect the problem that you are correctly facing. Do you mind to update to the most recent version of the code and perform the test again?
On the top of that, if you have the backtrace for the segfault it will be very useful. I did not managed to reproduce your problem. On c4fcb36f4cdd941fca413f85f7ff8603f257a3d9 I've made the debug logs a little bit more verbose about variable exclusion, you may want to share the debug logs as well.
@slabber commented on Mon Oct 23 2017
Still the same. CRS 3.0.2 with:
SecRuleUpdateTargetById 942421 "!REQUEST_COOKIES:/Example/"appendedThis is a backtrace from 351beb05676ab5dd5687057f38e7cb6c82f198c0
FYI, this is using https://github.com/kubernetes/ingress-nginx/tree/master/images/nginx-slim
@slabber commented on Mon Oct 23 2017
The last entry in the debug log at level 5 is:
There is no mention of the exception added with SecRuleUpdateTargetById
In this example, the appended lines were:
@zimmerle commented on Tue Oct 24 2017
Hi @slabber,
Thank you for the very detailed report. Should be fixed by 371fc03218efc205e8e42935c7d95b82c6047676. It will be great if you can confirm.
@slabber commented on Tue Oct 24 2017
Still segfault:
@slabber commented on Tue Oct 24 2017
I also still see nothing in the debug log for the exemptions:
I can also confirm that this is with 371fc03218efc205e8e42935c7d95b82c6047676
@zimmerle commented on Tue Oct 24 2017
@slabber are you using threads on nginx?
@slabber commented on Tue Oct 24 2017
Here's the build script from kubernetes ingress:
@slabber commented on Tue Oct 24 2017
So, yes. Threading is enabled by the build script from the kubernetes ingress guys.
@slabber commented on Wed Oct 25 2017
I have recompiled and retested without threads in nginx with exactly the same results and backtrace.
@victorhora commented on Sun Oct 29 2017
Hi @slabber
I tried to mimic your scenario with that build script to reproduce the issue and I think I'm pretty close:
By adding different combinations of the following:
I still couldn't reproduce this segfault.
But one thing I notice is that on the latest build I can't get SecRuleUpdateTargetById to work properly with regexes. But I could get working properly with
And I can see the following entry in the debug logs:
But do notice that you only get this entry on the debug logs if there's any matching rule ID about to be triggered. This means that, on PL3 or PL4 you would need to send a request like this:
It would be helpful if you could share:
a) The actual full HTTP request that triggers the segfault b) Full audit logs & Debug logs c) ModSecurity configuration file d) CRS configuration file (which Paranoia level you're running?) e) Nginx configuration file (which modules you have enabled?)
Thanks for the report.
@slabber commented on Mon Oct 30 2017
OK, here is a heavily redacted config and it won't work unless you create the certs needed for SSL. I haven't had a chance to check if this specific redacted version crashes or not but read the next comment for my strange findings:
@slabber commented on Mon Oct 30 2017
So with the above, the lines that seem to cause the segfault are the "if" sections both before and after the inclusion of the modsecurity configs. i.e.
and
If either of these are included then segfault still occurs. With both removed everything works as expected. Here is the curl line which is pretty much as you comment above:
@victorhora commented on Tue Oct 31 2017
Thanks @slabber
With this configuration file I can confirm the segfault occurring with the Nginx directive below enabled:
# enforce ssl on server side if ($pass_access_scheme = http) { return 301 https://$best_http_host$request_uri; }And a SecRuleUpdateTargetByID rule with any existing ID such as:
In this case any request triggers the segfault.
@slabber commented on Wed Nov 08 2017
@victorhora Any update on this? It's highly blocking for us kubernetes users. Even perhaps some update as to when you expect this to be fixed would be useful.
Thanks!