Closed md2k closed 7 years ago
Hello @md2k, could you please provide full nginx configuration (in recent versions it can be obtained by running nginx -T
), full modsecurity configuration, and backtraces?
Thanks.
I can semi-confirm this bug! Have already posted it in the modsec_dev list.
And also after updating the source from 12.12.16 - 27.01.17 the shared-collections are in /
root@waf-1-a-02:~# ls -la /etc/nginx/modsec/ -rw-r--r-- 1 root root 1048576 Jan 27 13:59 modsec-shared-collections -rw-r--r-- 1 root root 8192 Jan 27 14:00 modsec-shared-collections-lock
root@waf-1-a-02:~# ls -la / -rw-r--r-- 1 root root 1048576 Jan 27 14:34 modsec-shared-collections -rw-r--r-- 1 root root 8192 Jan 27 14:34 modsec-shared-collections-lock
I'm running latest N+ and build MS3 from source, not the N+ module.
I could also send you a nginx -T directly if you like cause it's very huge and sensible
@defanator i can send it, but it also big for me and and obfuscation will take some time (for nginx).
@md2k we can start from backtraces then.
@mimugmail if you're using N+, I would suggest to address this via N+ support channel. You can share backtraces here though.
It will be helpful if you can tell me how i can get backtrace from modsec/nginx
@defanator Since I'm not using the commercial WAF module I don't think that we can expect much support from N+, also I don't want to keep these guys rotating (was a promise to OwenGarret@Nginx) :)
I'll try to make a detailed bug report
@md2k you need to obtain core file and then use GDB to get output of the full bt
command. These links could be useful for further details:
https://www.nginx.com/resources/admin-guide/debug/ https://github.com/spiderlabs/modsecurity/tree/v3/master#debugging
Thank, will check them later, going to prepare virtual box for this to not mess with my production server
OS: Debian 8 Nginx: 1.11.9 MS3 source: 04.02.2017 Nginx connector: 04.02.2017
Backtrace full: root@nginx:~# gdb /opt/nginx/sbin/nginx /tmp/core GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /opt/nginx/sbin/nginx...done. [New LWP 16984] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `nginx: worker process'. Program terminated with signal SIGSEGV, Segmentation fault.
106 ../sysdeps/x86_64/strlen.S: No such file or directory. (gdb) backtrace full
No locals.
No symbol table info available.
at /usr/include/c++/4.9/bits/basic_string.h:2528
No locals.
fileName="/var/log/modsec_audit.log") at utils/shared_files.cc:42
current = 0x7fe960912000
at utils/shared_files.cc:181
a = <optimized out>
No locals.
No locals.
No locals.
No locals.
No locals.
t = 0x2454298
p = <optimized out>
n = <optimized out>
l = <optimized out>
c = 0x24542f8
i = <optimized out>
c = <optimized out>
worker = 0
---Type
i = 0
ch = {command = 1, pid = 0, slot = 0, fd = 0}
title = 0x2458ab4 "master process /opt/nginx/sbin/nginx"
p = <optimized out>
size = <optimized out>
i = <optimized out>
n = <optimized out>
sigio = <optimized out>
set = {__val = {0 <repeats 16 times>}}
itv = {it_interval = {tv_sec = 38111800, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 0}}
live = <optimized out>
delay = <optimized out>
ls = <optimized out>
ccf = 0x23e8878
b = <optimized out>
log = 0x6b9880 <ngx_log>
i = <optimized out>
cycle = 0x23e6d80
init_cycle = {conf_ctx = 0x0, pool = 0x23e6920, log = 0x6b9880 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, disk_full_time = 0,
handler = 0x0, data = 0x0, writer = 0x0, wdata = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0, free_connections = 0x0,
free_connection_n = 0, modules = 0x0, modules_n = 0, modules_used = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, reusable_connections_n = 0,
listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump = {
elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump_rbtree = {root = 0x0, sentinel = 0x0, insert = 0x0}, config_dump_sentinel = {
key = 0, left = 0x0, right = 0x0, parent = 0x0, color = 0 '\000', data = 0 '\000'}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0},
size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0},
connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 27, data = 0x23e6970 ""},
conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 17, data = 0x23e6970 ""}, prefix = {len = 12, data = 0x485bb0 "/opt/nginx//"}, lock_file = {
len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
cd = <optimized out>
---Type
nginx -T:
root@nginx:~# /opt/nginx/sbin/nginx -T nginx: the configuration file /opt/nginx//conf/nginx.conf syntax is ok nginx: configuration file /opt/nginx//conf/nginx.conf test is successful
worker_processes 1;
worker_rlimit_core 500M; working_directory /tmp/;
error_log logs/error.log debug;
pid logs/nginx.pid;
load_module "modules/ngx_http_modsecurity_module.so";
events { worker_connections 1024; }
http { include mime.types; default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
access_log logs/access.log;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
server {
listen 80;
server_name localhost;
modsecurity on;
modsecurity_rules_file /opt/nginx/modsec/main.conf;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
root html;
index index.html index.htm;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_index index.php;
# fastcgi_param SCRIPT_FILENAME /scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
# another virtual host using mix of IP-, name-, and port-based configuration
#
#server {
# listen 8000;
# listen somename:8080;
# server_name somename alias another.alias;
# location / {
# root html;
# index index.html index.htm;
# }
#}
# HTTPS server
#
#server {
# listen 443 ssl;
# server_name localhost;
# ssl_certificate cert.pem;
# ssl_certificate_key cert.key;
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 5m;
# ssl_ciphers HIGH:!aNULL:!MD5;
# ssl_prefer_server_ciphers on;
# location / {
# root html;
# index index.html index.htm;
# }
#}
}
types { text/html html htm shtml; text/css css; text/xml xml; image/gif gif; image/jpeg jpeg jpg; application/javascript js; application/atom+xml atom; application/rss+xml rss;
text/mathml mml;
text/plain txt;
text/vnd.sun.j2me.app-descriptor jad;
text/vnd.wap.wml wml;
text/x-component htc;
image/png png;
image/tiff tif tiff;
image/vnd.wap.wbmp wbmp;
image/x-icon ico;
image/x-jng jng;
image/x-ms-bmp bmp;
image/svg+xml svg svgz;
image/webp webp;
application/font-woff woff;
application/java-archive jar war ear;
application/json json;
application/mac-binhex40 hqx;
application/msword doc;
application/pdf pdf;
application/postscript ps eps ai;
application/rtf rtf;
application/vnd.apple.mpegurl m3u8;
application/vnd.ms-excel xls;
application/vnd.ms-fontobject eot;
application/vnd.ms-powerpoint ppt;
application/vnd.wap.wmlc wmlc;
application/vnd.google-earth.kml+xml kml;
application/vnd.google-earth.kmz kmz;
application/x-7z-compressed 7z;
application/x-cocoa cco;
application/x-java-archive-diff jardiff;
application/x-java-jnlp-file jnlp;
application/x-makeself run;
application/x-perl pl pm;
application/x-pilot prc pdb;
application/x-rar-compressed rar;
application/x-redhat-package-manager rpm;
application/x-sea sea;
application/x-shockwave-flash swf;
application/x-stuffit sit;
application/x-tcl tcl tk;
application/x-x509-ca-cert der pem crt;
application/x-xpinstall xpi;
application/xhtml+xml xhtml;
application/xspf+xml xspf;
application/zip zip;
application/octet-stream bin exe dll;
application/octet-stream deb;
application/octet-stream dmg;
application/octet-stream iso img;
application/octet-stream msi msp msm;
application/vnd.openxmlformats-officedocument.wordprocessingml.document docx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet xlsx;
application/vnd.openxmlformats-officedocument.presentationml.presentation pptx;
audio/midi mid midi kar;
audio/mpeg mp3;
audio/ogg ogg;
audio/x-m4a m4a;
audio/x-realaudio ra;
video/3gpp 3gpp 3gp;
video/mp2t ts;
video/mp4 mp4;
video/mpeg mpeg mpg;
video/quicktime mov;
video/webm webm;
video/x-flv flv;
video/x-m4v m4v;
video/x-mng mng;
video/x-ms-asf asx asf;
video/x-ms-wmv wmv;
video/x-msvideo avi;
}
This is just a development machine, non-productive!
Installed like in: http://www.routerperformance.net/howtos/setup-modsecurity-3-and-nginx-in-debian-8/
This is what nginx error logs throws after reload:
2017/02/04 22:03:11 [notice] 576#0: signal process started 2017/02/04 22:03:11 [notice] 574#0: signal 1 (SIGHUP) received, reconfiguring 2017/02/04 22:03:11 [debug] 574#0: wake up, sigio 0 2017/02/04 22:03:11 [notice] 574#0: reconfiguring 2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 00000000022EAD40:16384 @16 2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 0000000002CBF6A0:16384 @16 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002343720:4096 2017/02/04 22:03:11 [debug] 574#0: read: 21, 0000000002343720, 2833, 0 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 00000000022ECC08 2017/02/04 22:03:11 [debug] 574#0: module: ngx_http_modsecurity_module before ngx_http_range_header_filter_module:38 2017/02/04 22:03:11 [debug] 574#0: module: ngx_http_modsecurity_module i:48 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 00000000022EEA08 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 00000000022EEA80 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CC8330:4280 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CC93F0:4280 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CCA4B0:4280 2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 0000000002CCB570:16384 @16 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CCF580:4280 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD0640:4280 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD1700:4280 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD27C0:4096 2017/02/04 22:03:11 [debug] 574#0: include mime.types 2017/02/04 22:03:11 [debug] 574#0: include /opt/nginx//conf/mime.types 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD37D0:4096 2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 0000000002CD47E0:16384 @16 2017/02/04 22:03:11 [debug] 574#0: read: 32, 0000000002CD37D0, 3957, 0 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD87F0:4096 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 0000000002CD7C30 2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 0000000002CD9800:16384 @16 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 0000000002CD8760 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 0000000002CDB6E0 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:2048 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDE020:4352 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 0000000002CDBBB8 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD4550:512 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD4760:96 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024 2017/02/04 22:03:12 [debug] 575#0: epoll: fd:20 ev:2011 d:00007FF6D89140F0 2017/02/04 22:03:12 [debug] 575#0: epoll_wait() error on fd:20 ev:2011 2017/02/04 22:03:12 [debug] 575#0: channel handler 2017/02/04 22:03:12 [debug] 575#0: recvmsg() returned zero 2017/02/04 22:03:12 [debug] 575#0: channel: -1 2017/02/04 22:03:12 [debug] 575#0: epoll del connection: fd:20 2017/02/04 22:03:12 [debug] 575#0: reusable connection: 0 2017/02/04 22:03:12 [debug] 575#0: timer delta: 6870 2017/02/04 22:03:12 [debug] 575#0: worker cycle 2017/02/04 22:03:12 [debug] 575#0: epoll timer: -1 2017/02/04 22:03:43 [notice] 575#0: signal 15 (SIGTERM) received, exiting 2017/02/04 22:03:43 [info] 575#0: epoll_wait() failed (4: Interrupted system call) 2017/02/04 22:03:43 [debug] 575#0: timer delta: 31224 2017/02/04 22:03:43 [notice] 575#0: exiting 2017/02/04 22:03:43 [debug] 575#0: flush files 2017/02/04 22:03:43 [debug] 575#0: run cleanup: 0000000002358AE0 2017/02/04 22:03:43 [debug] 575#0: run cleanup: 00000000023547D0 2017/02/04 22:03:43 [debug] 575#0: cleanup resolver 2017/02/04 22:03:43 [debug] 575#0: run cleanup: 00000000023542F8
@md2k What distro do you use?
This is my last one ... also installed it with CentOS7 and also got the segfault.
@mimugmail, thanks for the backtrace!
It seems like the issue you're observing is related to a number of other issues we've been also facing. Latest attempt to provide a fix was made here: https://github.com/SpiderLabs/ModSecurity/pull/1306 (though there's a number of questions to that PR).
Tagging @zimmerle here - this is related to the libmodsecurity itself, not just the connector.
@md2k, were you able to grab a core and obtain a backtrace from it?
@defanator @zimmerle This error must be added to the source some time after 12.12.16, because my productive machine is running this code base and I can reload without any issues.
Hope you will find this one :)
@md2k install gdb packet, compile nginx --with-debug and add the stuff in the links @defanator added here. Then start nginx, reload it, then the segfault comes but there's no core, after that type a killall nginx and now there's the coredump int /tmp. Then type "gdb /opt/nginx/sbin/nginx /tmp/core" and "backtrace full"
@mimugmail thanks. I use Ubuntu 14.04.5 LTS with Nginx 1.10.2-1~trusty (from nginx repository)
@defanator most probably i will have time to deal with it next week due my main project workload.
As @zimmerle told me to disable lmdb in another issue I tried it on this one, but --without-lmdb I also get a segfault here.
Hi @defanator , as you asked, in Gist output of gdb
from core file after process crashed during nginx restart operation. (also configuration parameters which is used to compile nginx, its config (absolutely default, except modsec entries )), and log file from nginx with debug level
https://gist.github.com/md2k/4e18cc10649601bb93eed6a17bffc106
Thanks @md2k.
@defanator i added 2 more backtraces to same gist as Another GDB
(modsec compiled there with CFLAGS=-g -O0)
Hi,
The problem was related to shared memory. The first design was meant to be used by forked processes. In the forked process, the address space and file descriptor will be common in every forked-process (worker). The reload (in this case ./nginx -t) creates a new process, leading to different file descriptors and consequently a segfault. The segfault seems to be resolved, although I have faced `zombie' processes while stressing the implementation.
Do you guys mind to test? Those are the commits:
SharedFile class:
To use the SharedFile class within the DebugLogs:
The are available on the branch dev/parser:
Thank you guys, and sorry for the huge delay.
Thanks @zimmerle for fixing this! I can confirm that with dev parser branch I can cleanly compile and reload the process. BUT, now when I start nginx it looks like this:
root@nginx:~# /opt/nginx/sbin/nginx **root@nginx:~#
Also when reloading there are these asterisks. When I disable ModSec they are gone.
The performance like in the other issue is also not better than before, will post my results there.
Thank you!
Hello,
Same that @mimugmail with the last release (thanks for the update @zimmerle ) on check of status of nginx
Feb 24 12:29:59 ip-10-65-3-219 systemd[1]: Starting A high performance web server and a reverse proxy server...
Feb 24 12:30:03 ip-10-65-3-219 nginx[1062]: ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Feb 24 12:30:03 ip-10-65-3-219 nginx[1062]: **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Feb 24 12:30:07 ip-10-65-3-219 nginx[1067]: ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Feb 24 12:30:07 ip-10-65-3-219 systemd[1]: Started A high performance web server and a reverse proxy server.
Feb 24 12:30:07 ip-10-65-3-219 nginx[1067]: **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
@zimmerle I've just tested libmodsecurity from the current head of v3/dev/parser, and can confirm that nginx is not segfaulting anymore.
Then I realized that I've been running nginx with older connector module, built with libmodsecurity v3/master. I tried to rebuild it with v3/dev/parser code and got the following:
cc -c -fPIC -pipe -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include -I objs -I src/http -I src/http/modules \
-o objs/addon/src/ngx_http_modsecurity_module.o \
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c: In function 'ngx_http_modsecurity_create_main_conf':
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:473:34: error: passing argument 2 of 'msc_set_log_cb' from incompatible pointer type [-Werror=incompatible-pointer-types]
msc_set_log_cb(conf->modsec, ngx_http_modsecurity_log);
^~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../ModSecurity-nginx/src/ngx_http_modsecurity_common.h:25:0,
from ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:21:
/usr/include/modsecurity/modsecurity.h:325:6: note: expected 'ModSecLogCb {aka void (*)(void *, const void *)}' but argument is of type 'void (*)(void *, const char *)'
void msc_set_log_cb(ModSecurity *msc, ModSecLogCb cb);
^~~~~~~~~~~~~~
cc1: all warnings being treated as errors
Are there any plans to merge v3/dev/parser to v3/master, or cherry pick a set of changes affecting the logging part, so we could adjust nginx connector code here?
v3/dev/parser is now part of v3/master ;)
Thank you for the reports ;)
I got this error too
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c: In function 'ngx_http_modsecurity_create_main_conf':
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:426:5: error: passing argument 2 of 'msc_set_log_cb' from incompatible pointer type [-Werror]
msc_set_log_cb(conf->modsec, ngx_http_modsecurity_log);
^
In file included from ../ModSecurity-nginx/src/ngx_http_modsecurity_common.h:25:0,
from ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:21:
/tmp/ModSecurity/headers/modsecurity/modsecurity.h:325:6: note: expected 'ModSecLogCb' but argument is of type 'void (*)(void *, const char *)'
void msc_set_log_cb(ModSecurity *msc, ModSecLogCb cb);
^
cc1: all warnings being treated as errors
What I am doing wrong or how do I fix it? Thanks.
Hi @hernandanielg,
Please upgrade both: ModSecurity and ModSecurity-nginx connector
Worked like a charm :) thanks!
I am having this issue now; details here:
https://github.com/SpiderLabs/ModSecurity/issues/1318#issuecomment-295539544
any thoughts?
Hi Devs, i noticed that latest master of connector (maybe issue not in connector exactly) causing nginx segfaults during its restart. Reload or Stop then Start do not cause such segfaults. Also by default files for shared collection dropped to root folder '/'
But, during restart Nginx with init/init.d scripts those files created in directory where cli command was used.