md2k commented 7 years ago

Hi Devs, i noticed that latest master of connector (maybe issue not in connector exactly) causing nginx segfaults during its restart. Reload or Stop then Start do not cause such segfaults. Also by default files for shared collection dropped to root folder '/'

ls -l / | grep shared
-rw-r--r--   1 root root 1048576 Jan 13 17:03 modsec-shared-collections
-rw-r--r--   1 root root    8192 Jan 13 17:05 modsec-shared-collections-lock

But, during restart Nginx with init/init.d scripts those files created in directory where cli command was used.

Jan 13 17:03:07 abyss kernel: [15515838.077384] nginx[7190]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077395] nginx[7192]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077398] nginx[7186]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077400] nginx[7191]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077401]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.077402] nginx[7188]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b9208 error 4
Jan 13 17:03:07 abyss kernel: [15515838.077403]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.077404]
Jan 13 17:03:07 abyss kernel: [15515838.077405]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.077405]
Jan 13 17:03:07 abyss kernel: [15515838.077406]
Jan 13 17:03:07 abyss kernel: [15515838.077407]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.077414]  in libc-2.19.so[7fbe74bd3000+1ba000]
Jan 13 17:03:07 abyss kernel: [15515838.080151] nginx[7185]: segfault at 7f9548807000 ip 00007fbe74c5b9da sp 00007ffcde6b92e8 error 4 in libc-2.19.so[7fbe74bd3000+1ba000]

Jan 13 17:03:24 abyss kernel: [15515854.689932] nginx[7689]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4
Jan 13 17:03:24 abyss kernel: [15515854.689934] nginx[7687]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.689936]  in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.689960] nginx[7686]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.689978] nginx[7690]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.689983] nginx[7691]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.690068] nginx[7688]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954ecc8 error 4 in libc-2.19.so[7fea863e5000+1ba000]
Jan 13 17:03:24 abyss kernel: [15515854.692350] nginx[7685]: segfault at 7fc7b8ed5000 ip 00007fea8646d9da sp 00007ffe7954eda8 error 4 in libc-2.19.so[7fea863e5000+1ba000]

defanator commented 7 years ago

Hello @md2k, could you please provide full nginx configuration (in recent versions it can be obtained by running nginx -T), full modsecurity configuration, and backtraces?

Thanks.

mimugmail commented 7 years ago

I can semi-confirm this bug! Have already posted it in the modsec_dev list.

And also after updating the source from 12.12.16 - 27.01.17 the shared-collections are in /

root@waf-1-a-02:~# ls -la /etc/nginx/modsec/ -rw-r--r-- 1 root root 1048576 Jan 27 13:59 modsec-shared-collections -rw-r--r-- 1 root root 8192 Jan 27 14:00 modsec-shared-collections-lock

root@waf-1-a-02:~# ls -la / -rw-r--r-- 1 root root 1048576 Jan 27 14:34 modsec-shared-collections -rw-r--r-- 1 root root 8192 Jan 27 14:34 modsec-shared-collections-lock

I'm running latest N+ and build MS3 from source, not the N+ module.

mimugmail commented 7 years ago

I could also send you a nginx -T directly if you like cause it's very huge and sensible

md2k commented 7 years ago

@defanator i can send it, but it also big for me and and obfuscation will take some time (for nginx).

defanator commented 7 years ago

@md2k we can start from backtraces then.

defanator commented 7 years ago

@mimugmail if you're using N+, I would suggest to address this via N+ support channel. You can share backtraces here though.

md2k commented 7 years ago

It will be helpful if you can tell me how i can get backtrace from modsec/nginx

mimugmail commented 7 years ago

@defanator Since I'm not using the commercial WAF module I don't think that we can expect much support from N+, also I don't want to keep these guys rotating (was a promise to OwenGarret@Nginx) :)

I'll try to make a detailed bug report

defanator commented 7 years ago

@md2k you need to obtain core file and then use GDB to get output of the full bt command. These links could be useful for further details:

https://www.nginx.com/resources/admin-guide/debug/ https://github.com/spiderlabs/modsecurity/tree/v3/master#debugging

md2k commented 7 years ago

Thank, will check them later, going to prepare virtual box for this to not mess with my production server

mimugmail commented 7 years ago

OS: Debian 8 Nginx: 1.11.9 MS3 source: 04.02.2017 Nginx connector: 04.02.2017

Backtrace full: root@nginx:~# gdb /opt/nginx/sbin/nginx /tmp/core GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: http://www.gnu.org/software/gdb/bugs/. Find the GDB manual and other documentation resources online at: http://www.gnu.org/software/gdb/documentation/. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /opt/nginx/sbin/nginx...done. [New LWP 16984] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". Core was generated by `nginx: worker process'. Program terminated with signal SIGSEGV, Segmentation fault.

0 strlen () at ../sysdeps/x86_64/strlen.S:106

106 ../sysdeps/x86_64/strlen.S: No such file or directory. (gdb) backtrace full

0 strlen () at ../sysdeps/x86_64/strlen.S:106

No locals.

1 0x00007fe95e437c28 in std::string::compare(char const*) const () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

No symbol table info available.

2 0x00007fe95f5d36cc in operator==<char, std::char_traits, std::allocator > (__rhs="/var/log/modsec_audit.log", __lhs=)

at /usr/include/c++/4.9/bits/basic_string.h:2528

No locals.

3 modsecurity::utils::SharedFiles::find_handler (this=this@entry=0x7fe95f857ce8 <modsecurity::utils::SharedFiles::getInstance()::instance>,

fileName="/var/log/modsec_audit.log") at utils/shared_files.cc:42
    current = 0x7fe960912000

4 0x00007fe95f5d3710 in modsecurity::utils::SharedFiles::close (this=0x7fe95f857ce8 <modsecurity::utils::SharedFiles::getInstance()::instance>, fileName=...)

at utils/shared_files.cc:181
    a = <optimized out>

5 0x00007fe95f56d119 in modsecurity::audit_log::writer::Serial::~Serial (this=0x2443220, __in_chrg=) at audit_log/writer/serial.cc:28

No locals.

6 0x00007fe95f56b148 in modsecurity::audit_log::AuditLog::~AuditLog (this=0x2443100, __in_chrg=) at audit_log/audit_log.cc:69

No locals.

7 0x00007fe95f56fc16 in ~RulesProperties (this=0x24471a0, __in_chrg=) at ../headers/modsecurity/rules_properties.h:106

No locals.

8 modsecurity::Rules::~Rules (this=0x24471a0, __in_chrg=) at rules.cc:80

No locals.

9 0x00007fe95f56ff0e in modsecurity::msc_rules_cleanup (rules=0x24471a0) at rules.cc:335

No locals.

10 0x00007fe95f85a4d4 in ngx_http_modsecurity_config_cleanup (data=0x2454298) at /opt/ModSecurity-nginx/src/ngx_http_modsecurity_module.c:595

    t = 0x2454298

11 0x000000000040df31 in ngx_destroy_pool (pool=0x23e6d30) at src/core/ngx_palloc.c:57

    p = <optimized out>
    n = <optimized out>
    l = <optimized out>
    c = 0x24542f8

12 0x00000000004330a0 in ngx_worker_process_exit (cycle=cycle@entry=0x23e6d80) at src/os/unix/ngx_process_cycle.c:1001

    i = <optimized out>
    c = <optimized out>

13 0x00000000004331af in ngx_worker_process_cycle (cycle=cycle@entry=0x23e6d80, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:758

    worker = 0

14 0x000000000043185a in ngx_spawn_process (cycle=cycle@entry=0x23e6d80, proc=proc@entry=0x4330d8 , data=data@entry=0x0,

---Type to continue, or q to quit--- name=name@entry=0x48ad7e "worker process", respawn=respawn@entry=-3) at src/os/unix/ngx_process.c:198 on = 1 pid = 0 s = 0

15 0x000000000043241d in ngx_start_worker_processes (cycle=cycle@entry=0x23e6d80, n=1, type=type@entry=-3) at src/os/unix/ngx_process_cycle.c:358

    i = 0
    ch = {command = 1, pid = 0, slot = 0, fd = 0}

16 0x0000000000433a7b in ngx_master_process_cycle (cycle=cycle@entry=0x23e6d80) at src/os/unix/ngx_process_cycle.c:130

    title = 0x2458ab4 "master process /opt/nginx/sbin/nginx"
    p = <optimized out>
    size = <optimized out>
    i = <optimized out>
    n = <optimized out>
    sigio = <optimized out>
    set = {__val = {0 <repeats 16 times>}}
    itv = {it_interval = {tv_sec = 38111800, tv_usec = 0}, it_value = {tv_sec = 0, tv_usec = 0}}
    live = <optimized out>
    delay = <optimized out>
    ls = <optimized out>
    ccf = 0x23e8878

17 0x000000000040c7b7 in main (argc=, argv=) at src/core/nginx.c:368

    b = <optimized out>
    log = 0x6b9880 <ngx_log>
    i = <optimized out>
    cycle = 0x23e6d80
    init_cycle = {conf_ctx = 0x0, pool = 0x23e6920, log = 0x6b9880 <ngx_log>, new_log = {log_level = 0, file = 0x0, connection = 0, disk_full_time = 0,
        handler = 0x0, data = 0x0, writer = 0x0, wdata = 0x0, action = 0x0, next = 0x0}, log_use_stderr = 0, files = 0x0, free_connections = 0x0,
      free_connection_n = 0, modules = 0x0, modules_n = 0, modules_used = 0, reusable_connections_queue = {prev = 0x0, next = 0x0}, reusable_connections_n = 0,
      listening = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, paths = {elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump = {
        elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0}, config_dump_rbtree = {root = 0x0, sentinel = 0x0, insert = 0x0}, config_dump_sentinel = {
        key = 0, left = 0x0, right = 0x0, parent = 0x0, color = 0 '\000', data = 0 '\000'}, open_files = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0},
        size = 0, nalloc = 0, pool = 0x0}, shared_memory = {last = 0x0, part = {elts = 0x0, nelts = 0, next = 0x0}, size = 0, nalloc = 0, pool = 0x0},
      connection_n = 0, files_n = 0, connections = 0x0, read_events = 0x0, write_events = 0x0, old_cycle = 0x0, conf_file = {len = 27, data = 0x23e6970 ""},
      conf_param = {len = 0, data = 0x0}, conf_prefix = {len = 17, data = 0x23e6970 ""}, prefix = {len = 12, data = 0x485bb0 "/opt/nginx//"}, lock_file = {
        len = 0, data = 0x0}, hostname = {len = 0, data = 0x0}}
    cd = <optimized out>

---Type to continue, or q to quit--- ccf =

nginx -T:

root@nginx:~# /opt/nginx/sbin/nginx -T nginx: the configuration file /opt/nginx//conf/nginx.conf syntax is ok nginx: configuration file /opt/nginx//conf/nginx.conf test is successful

configuration file /opt/nginx//conf/nginx.conf:

user nobody;

worker_processes 1;

worker_rlimit_core 500M; working_directory /tmp/;

error_log logs/error.log;

error_log logs/error.log notice;

error_log logs/error.log debug;

pid logs/nginx.pid;

load_module "modules/ngx_http_modsecurity_module.so";

events { worker_connections 1024; }

http { include mime.types; default_type application/octet-stream;

#log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
#                  '$status $body_bytes_sent "$http_referer" '
#                  '"$http_user_agent" "$http_x_forwarded_for"';

access_log  logs/access.log;

sendfile        on;
#tcp_nopush     on;

#keepalive_timeout  0;
keepalive_timeout  65;

#gzip  on;

server {
    listen       80;
    server_name  localhost;

modsecurity on;
modsecurity_rules_file /opt/nginx/modsec/main.conf;

    #charset koi8-r;

    #access_log  logs/host.access.log  main;

    location / {
        root   html;
        index  index.html index.htm;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    #location ~ \.php$ {
    #    root           html;
    #    fastcgi_pass   127.0.0.1:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}

# another virtual host using mix of IP-, name-, and port-based configuration
#
#server {
#    listen       8000;
#    listen       somename:8080;
#    server_name  somename  alias  another.alias;

#    location / {
#        root   html;
#        index  index.html index.htm;
#    }
#}

# HTTPS server
#
#server {
#    listen       443 ssl;
#    server_name  localhost;

#    ssl_certificate      cert.pem;
#    ssl_certificate_key  cert.key;

#    ssl_session_cache    shared:SSL:1m;
#    ssl_session_timeout  5m;

#    ssl_ciphers  HIGH:!aNULL:!MD5;
#    ssl_prefer_server_ciphers  on;

#    location / {
#        root   html;
#        index  index.html index.htm;
#    }
#}

}

configuration file /opt/nginx//conf/mime.types:

types { text/html html htm shtml; text/css css; text/xml xml; image/gif gif; image/jpeg jpeg jpg; application/javascript js; application/atom+xml atom; application/rss+xml rss;

text/mathml                           mml;
text/plain                            txt;
text/vnd.sun.j2me.app-descriptor      jad;
text/vnd.wap.wml                      wml;
text/x-component                      htc;

image/png                             png;
image/tiff                            tif tiff;
image/vnd.wap.wbmp                    wbmp;
image/x-icon                          ico;
image/x-jng                           jng;
image/x-ms-bmp                        bmp;
image/svg+xml                         svg svgz;
image/webp                            webp;

application/font-woff                 woff;
application/java-archive              jar war ear;
application/json                      json;
application/mac-binhex40              hqx;
application/msword                    doc;
application/pdf                       pdf;
application/postscript                ps eps ai;
application/rtf                       rtf;
application/vnd.apple.mpegurl         m3u8;
application/vnd.ms-excel              xls;
application/vnd.ms-fontobject         eot;
application/vnd.ms-powerpoint         ppt;
application/vnd.wap.wmlc              wmlc;
application/vnd.google-earth.kml+xml  kml;
application/vnd.google-earth.kmz      kmz;
application/x-7z-compressed           7z;
application/x-cocoa                   cco;
application/x-java-archive-diff       jardiff;
application/x-java-jnlp-file          jnlp;
application/x-makeself                run;
application/x-perl                    pl pm;
application/x-pilot                   prc pdb;
application/x-rar-compressed          rar;
application/x-redhat-package-manager  rpm;
application/x-sea                     sea;
application/x-shockwave-flash         swf;
application/x-stuffit                 sit;
application/x-tcl                     tcl tk;
application/x-x509-ca-cert            der pem crt;
application/x-xpinstall               xpi;
application/xhtml+xml                 xhtml;
application/xspf+xml                  xspf;
application/zip                       zip;

application/octet-stream              bin exe dll;
application/octet-stream              deb;
application/octet-stream              dmg;
application/octet-stream              iso img;
application/octet-stream              msi msp msm;

application/vnd.openxmlformats-officedocument.wordprocessingml.document    docx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet          xlsx;
application/vnd.openxmlformats-officedocument.presentationml.presentation  pptx;

audio/midi                            mid midi kar;
audio/mpeg                            mp3;
audio/ogg                             ogg;
audio/x-m4a                           m4a;
audio/x-realaudio                     ra;

video/3gpp                            3gpp 3gp;
video/mp2t                            ts;
video/mp4                             mp4;
video/mpeg                            mpeg mpg;
video/quicktime                       mov;
video/webm                            webm;
video/x-flv                           flv;
video/x-m4v                           m4v;
video/x-mng                           mng;
video/x-ms-asf                        asx asf;
video/x-ms-wmv                        wmv;
video/x-msvideo                       avi;

}

This is just a development machine, non-productive!

Installed like in: http://www.routerperformance.net/howtos/setup-modsecurity-3-and-nginx-in-debian-8/

mimugmail commented 7 years ago

This is what nginx error logs throws after reload:

2017/02/04 22:03:11 [notice] 576#0: signal process started 2017/02/04 22:03:11 [notice] 574#0: signal 1 (SIGHUP) received, reconfiguring 2017/02/04 22:03:11 [debug] 574#0: wake up, sigio 0 2017/02/04 22:03:11 [notice] 574#0: reconfiguring 2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 00000000022EAD40:16384 @16 2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 0000000002CBF6A0:16384 @16 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002343720:4096 2017/02/04 22:03:11 [debug] 574#0: read: 21, 0000000002343720, 2833, 0 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 00000000022ECC08 2017/02/04 22:03:11 [debug] 574#0: module: ngx_http_modsecurity_module before ngx_http_range_header_filter_module:38 2017/02/04 22:03:11 [debug] 574#0: module: ngx_http_modsecurity_module i:48 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 00000000022EEA08 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 00000000022EEA80 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CC8330:4280 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CC93F0:4280 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CCA4B0:4280 2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 0000000002CCB570:16384 @16 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CCF580:4280 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD0640:4280 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD1700:4280 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD27C0:4096 2017/02/04 22:03:11 [debug] 574#0: include mime.types 2017/02/04 22:03:11 [debug] 574#0: include /opt/nginx//conf/mime.types 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD37D0:4096 2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 0000000002CD47E0:16384 @16 2017/02/04 22:03:11 [debug] 574#0: read: 32, 0000000002CD37D0, 3957, 0 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD87F0:4096 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 0000000002CD7C30 2017/02/04 22:03:11 [debug] 574#0: posix_memalign: 0000000002CD9800:16384 @16 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 0000000002CD8760 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 0000000002CDB6E0 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:2048 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDE020:4352 2017/02/04 22:03:11 [debug] 574#0: add cleanup: 0000000002CDBBB8 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD4550:512 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CD4760:96 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024 2017/02/04 22:03:11 [debug] 574#0: malloc: 0000000002CDD810:1024 2017/02/04 22:03:12 [debug] 575#0: epoll: fd:20 ev:2011 d:00007FF6D89140F0 2017/02/04 22:03:12 [debug] 575#0: epoll_wait() error on fd:20 ev:2011 2017/02/04 22:03:12 [debug] 575#0: channel handler 2017/02/04 22:03:12 [debug] 575#0: recvmsg() returned zero 2017/02/04 22:03:12 [debug] 575#0: channel: -1 2017/02/04 22:03:12 [debug] 575#0: epoll del connection: fd:20 2017/02/04 22:03:12 [debug] 575#0: reusable connection: 0 2017/02/04 22:03:12 [debug] 575#0: timer delta: 6870 2017/02/04 22:03:12 [debug] 575#0: worker cycle 2017/02/04 22:03:12 [debug] 575#0: epoll timer: -1 2017/02/04 22:03:43 [notice] 575#0: signal 15 (SIGTERM) received, exiting 2017/02/04 22:03:43 [info] 575#0: epoll_wait() failed (4: Interrupted system call) 2017/02/04 22:03:43 [debug] 575#0: timer delta: 31224 2017/02/04 22:03:43 [notice] 575#0: exiting 2017/02/04 22:03:43 [debug] 575#0: flush files 2017/02/04 22:03:43 [debug] 575#0: run cleanup: 0000000002358AE0 2017/02/04 22:03:43 [debug] 575#0: run cleanup: 00000000023547D0 2017/02/04 22:03:43 [debug] 575#0: cleanup resolver 2017/02/04 22:03:43 [debug] 575#0: run cleanup: 00000000023542F8

@md2k What distro do you use?

mimugmail commented 7 years ago

This is my last one ... also installed it with CentOS7 and also got the segfault.

defanator commented 7 years ago

@mimugmail, thanks for the backtrace!

It seems like the issue you're observing is related to a number of other issues we've been also facing. Latest attempt to provide a fix was made here: https://github.com/SpiderLabs/ModSecurity/pull/1306 (though there's a number of questions to that PR).

Tagging @zimmerle here - this is related to the libmodsecurity itself, not just the connector.

@md2k, were you able to grab a core and obtain a backtrace from it?

mimugmail commented 7 years ago

@defanator @zimmerle This error must be added to the source some time after 12.12.16, because my productive machine is running this code base and I can reload without any issues.

Hope you will find this one :)

@md2k install gdb packet, compile nginx --with-debug and add the stuff in the links @defanator added here. Then start nginx, reload it, then the segfault comes but there's no core, after that type a killall nginx and now there's the coredump int /tmp. Then type "gdb /opt/nginx/sbin/nginx /tmp/core" and "backtrace full"

md2k commented 7 years ago

@mimugmail thanks. I use Ubuntu 14.04.5 LTS with Nginx 1.10.2-1~trusty (from nginx repository)

@defanator most probably i will have time to deal with it next week due my main project workload.

mimugmail commented 7 years ago

As @zimmerle told me to disable lmdb in another issue I tried it on this one, but --without-lmdb I also get a segfault here.

md2k commented 7 years ago

Hi @defanator , as you asked, in Gist output of gdb from core file after process crashed during nginx restart operation. (also configuration parameters which is used to compile nginx, its config (absolutely default, except modsec entries )), and log file from nginx with debug level https://gist.github.com/md2k/4e18cc10649601bb93eed6a17bffc106

defanator commented 7 years ago

Thanks @md2k.

md2k commented 7 years ago

@defanator i added 2 more backtraces to same gist as Another GDB (modsec compiled there with CFLAGS=-g -O0)

zimmerle commented 7 years ago

Hi,

The problem was related to shared memory. The first design was meant to be used by forked processes. In the forked process, the address space and file descriptor will be common in every forked-process (worker). The reload (in this case ./nginx -t) creates a new process, leading to different file descriptors and consequently a segfault. The segfault seems to be resolved, although I have faced `zombie' processes while stressing the implementation.

Do you guys mind to test? Those are the commits:

SharedFile class:

To use the SharedFile class within the DebugLogs:

https://github.com/SpiderLabs/ModSecurity/commit/1225076d3a81732ec7ba6ed553335387d4110024

The are available on the branch dev/parser:

https://github.com/SpiderLabs/ModSecurity/tree/v3/dev/parser

Thank you guys, and sorry for the huge delay.

mimugmail commented 7 years ago

Thanks @zimmerle for fixing this! I can confirm that with dev parser branch I can cleanly compile and reload the process. BUT, now when I start nginx it looks like this:

root@nginx:~# /opt/nginx/sbin/nginx **root@nginx:~#

Also when reloading there are these asterisks. When I disable ModSec they are gone.

The performance like in the other issue is also not better than before, will post my results there.

Thank you!

Menahem1 commented 7 years ago

Hello,

Same that @mimugmail with the last release (thanks for the update @zimmerle ) on check of status of nginx

Feb 24 12:29:59 ip-10-65-3-219 systemd[1]: Starting A high performance web server and a reverse proxy server...
Feb 24 12:30:03 ip-10-65-3-219 nginx[1062]: ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Feb 24 12:30:03 ip-10-65-3-219 nginx[1062]: **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Feb 24 12:30:07 ip-10-65-3-219 nginx[1067]: ********************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************
Feb 24 12:30:07 ip-10-65-3-219 systemd[1]: Started A high performance web server and a reverse proxy server.
Feb 24 12:30:07 ip-10-65-3-219 nginx[1067]: **************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************************

defanator commented 7 years ago

@zimmerle I've just tested libmodsecurity from the current head of v3/dev/parser, and can confirm that nginx is not segfaulting anymore.

Then I realized that I've been running nginx with older connector module, built with libmodsecurity v3/master. I tried to rebuild it with v3/dev/parser code and got the following:

cc -c -fPIC -pipe  -O -W -Wall -Wpointer-arith -Wno-unused-parameter -Werror -g  -I src/core -I src/event -I src/event/modules -I src/os/unix -I /usr/include -I objs -I src/http -I src/http/modules \
    -o objs/addon/src/ngx_http_modsecurity_module.o \
    ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c: In function 'ngx_http_modsecurity_create_main_conf':
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:473:34: error: passing argument 2 of 'msc_set_log_cb' from incompatible pointer type [-Werror=incompatible-pointer-types]
     msc_set_log_cb(conf->modsec, ngx_http_modsecurity_log);
                                  ^~~~~~~~~~~~~~~~~~~~~~~~
In file included from ../ModSecurity-nginx/src/ngx_http_modsecurity_common.h:25:0,
                 from ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:21:
/usr/include/modsecurity/modsecurity.h:325:6: note: expected 'ModSecLogCb {aka void (*)(void *, const void *)}' but argument is of type 'void (*)(void *, const char *)'
 void msc_set_log_cb(ModSecurity *msc, ModSecLogCb cb);
      ^~~~~~~~~~~~~~
cc1: all warnings being treated as errors

Are there any plans to merge v3/dev/parser to v3/master, or cherry pick a set of changes affecting the logging part, so we could adjust nginx connector code here?

zimmerle commented 7 years ago

v3/dev/parser is now part of v3/master ;)

Thank you for the reports ;)

hernandanielg commented 7 years ago

I got this error too

../ModSecurity-nginx/src/ngx_http_modsecurity_module.c: In function 'ngx_http_modsecurity_create_main_conf':
../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:426:5: error: passing argument 2 of 'msc_set_log_cb' from incompatible pointer type [-Werror]
     msc_set_log_cb(conf->modsec, ngx_http_modsecurity_log);
     ^
In file included from ../ModSecurity-nginx/src/ngx_http_modsecurity_common.h:25:0,
                 from ../ModSecurity-nginx/src/ngx_http_modsecurity_module.c:21:
/tmp/ModSecurity/headers/modsecurity/modsecurity.h:325:6: note: expected 'ModSecLogCb' but argument is of type 'void (*)(void *, const char *)'
 void msc_set_log_cb(ModSecurity *msc, ModSecLogCb cb);
      ^
cc1: all warnings being treated as errors

What I am doing wrong or how do I fix it? Thanks.

zimmerle commented 7 years ago

Hi @hernandanielg,

Please upgrade both: ModSecurity and ModSecurity-nginx connector

hernandanielg commented 7 years ago

Worked like a charm :) thanks!

jurgenweber commented 7 years ago

I am having this issue now; details here:

https://github.com/SpiderLabs/ModSecurity/issues/1318#issuecomment-295539544

any thoughts?

owasp-modsecurity / ModSecurity-nginx

Nginx Restart with loaded module cause segfaults (maybe problem for ModSec v3 and not connector itself) #29

0 strlen () at ../sysdeps/x86_64/strlen.S:106

0 strlen () at ../sysdeps/x86_64/strlen.S:106

1 0x00007fe95e437c28 in std::string::compare(char const*) const () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6

2 0x00007fe95f5d36cc in operator==<char, std::char_traits, std::allocator > (__rhs="/var/log/modsec_audit.log", __lhs=)

3 modsecurity::utils::SharedFiles::find_handler (this=this@entry=0x7fe95f857ce8 <modsecurity::utils::SharedFiles::getInstance()::instance>,

4 0x00007fe95f5d3710 in modsecurity::utils::SharedFiles::close (this=0x7fe95f857ce8 <modsecurity::utils::SharedFiles::getInstance()::instance>, fileName=...)

5 0x00007fe95f56d119 in modsecurity::audit_log::writer::Serial::~Serial (this=0x2443220, __in_chrg=) at audit_log/writer/serial.cc:28

6 0x00007fe95f56b148 in modsecurity::audit_log::AuditLog::~AuditLog (this=0x2443100, __in_chrg=) at audit_log/audit_log.cc:69

7 0x00007fe95f56fc16 in ~RulesProperties (this=0x24471a0, __in_chrg=) at ../headers/modsecurity/rules_properties.h:106

8 modsecurity::Rules::~Rules (this=0x24471a0, __in_chrg=) at rules.cc:80

9 0x00007fe95f56ff0e in modsecurity::msc_rules_cleanup (rules=0x24471a0) at rules.cc:335

10 0x00007fe95f85a4d4 in ngx_http_modsecurity_config_cleanup (data=0x2454298) at /opt/ModSecurity-nginx/src/ngx_http_modsecurity_module.c:595

11 0x000000000040df31 in ngx_destroy_pool (pool=0x23e6d30) at src/core/ngx_palloc.c:57

12 0x00000000004330a0 in ngx_worker_process_exit (cycle=cycle@entry=0x23e6d80) at src/os/unix/ngx_process_cycle.c:1001

13 0x00000000004331af in ngx_worker_process_cycle (cycle=cycle@entry=0x23e6d80, data=data@entry=0x0) at src/os/unix/ngx_process_cycle.c:758

14 0x000000000043185a in ngx_spawn_process (cycle=cycle@entry=0x23e6d80, proc=proc@entry=0x4330d8 , data=data@entry=0x0,

15 0x000000000043241d in ngx_start_worker_processes (cycle=cycle@entry=0x23e6d80, n=1, type=type@entry=-3) at src/os/unix/ngx_process_cycle.c:358

16 0x0000000000433a7b in ngx_master_process_cycle (cycle=cycle@entry=0x23e6d80) at src/os/unix/ngx_process_cycle.c:130

17 0x000000000040c7b7 in main (argc=, argv=) at src/core/nginx.c:368

configuration file /opt/nginx//conf/nginx.conf:

user nobody;

error_log logs/error.log;

error_log logs/error.log notice;

configuration file /opt/nginx//conf/mime.types: