worker process exited on signal 11 in Learning Mode on Ubuntu 12.04 64bit

[GoogleCodeExporter]

I will add more details if needed. Manual compilation and upgrade nginx or 
naxsi don't help. Files permissions loooks good. I thought about reporting the 
problem to the creator of nginx, but because the problem occurs only with naxsi 
I think this is better place. 

What steps will reproduce the problem?
1. Run nginx-naxsi in learning mode on Ubuntu 64bit (nginx as reverse-proxy, 
using ssl).
2. Send some request which will be noticed by naxsi (not "blocked" because of 
learning mode), 
for example: 
http://example.com/login/?user=|"`id`"|
https://example.com/sipsys/users/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%
ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f
0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%
80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae
%f0%80%80%ae/etc/passwd

3. Look into logs:
Sometimes it happens once in hundreds of queries, sometimes several times,  
nginx worker is killed, from /var/log/nginx/error.log:
2012/10/15 15:44:47 [alert] 4994 # 0: worker process 4995 exited on signal 11
2012/10/15 15:50:40 [alert] 4994 # 0: worker process 5123 exited on signal 11

Coredump gives:
#0  0x000000000040881d in ngx_hash_find (hash=<optimized out>, key=<optimized 
out>, name=0xa3e696c2f3c0a3e <Address 0xa3e696c2f3c0a3e out of bounds>, len=9) 
at src/core/ngx_hash.c:34
34  src/core/ngx_hash.c: No such file or directory.

Site error logs looks correctly:
2012/10/15 15:44:47 [error] 4995#0: *3 NAXSI_FMT: 
ip=10.0.0.8&server=example.com&uri=/login/&total_processed=2&total_blocked=2&zon
e0=ARGS&id0=1001&var_name0=user&zone1=ARGS&id1=1005&var_name1=user&zone2=ARGS&id
2=1314&var_name2=user, client: 10.0.0.8, server: example.com, request: "GET 
/login/?user=|%22`id`%22| HTTP/1.1", host: "example.com"

It happens only in Learning Mode, only when queries is triggering an alert. 
Request the above allways is correctly blocked by naxsi, but results in an 
error only sometimes (but more often than for example "example.com?a=<>"). I 
got it several times after restarting nginx, query 
(example.com/login?user=|"`id`"|) once resulted in a error (in logs nginx: 
worker process exited on signal # 11, on browser: page does not return the 
contents), and all further work correctly (in my case, redirect to the home 
page, because I do not have the "login").

What is the expected output? What do you see instead?
Nginx workers should not exit with error code 11 :)

What version of the product are you using? On what operating system?
-Ubuntu 12.03 "precise" 64bit, kernel 3.2.0-31-virtual (tested on 3 machines 
with similar but different system configuration)
Tested with the same result on:
nginx 1.1.19 with naxsi 0.44 (ubuntu repository)
nginx 1.1.19 with naxsi 0.48
nginx 1.2.4 with naxsi 0.48 (dotdeb repository)
currently: 
nginx version: nginx/1.2.4
TLS SNI support enabled
configure arguments: --prefix=/usr/share/nginx 
--conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log 
--http-client-body-temp-path=/var/lib/nginx/body 
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi 
--http-log-path=/var/log/nginx/access.log 
--http-proxy-temp-path=/var/lib/nginx/proxy 
--http-scgi-temp-path=/var/lib/nginx/scgi 
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi --lock-path=/var/lock/nginx.lock 
--pid-path=/var/run/nginx.pid --with-pcre-jit --with-file-aio 
--with-http_gzip_static_module --with-http_ssl_module 
--without-mail_pop3_module --without-mail_smtp_module 
--without-mail_imap_module --without-http_uwsgi_module 
--without-http_scgi_module --with-ipv6 --with-http_stub_status_module 
--add-module=/usr/src/nginx/source/nginx-1.2.4/debian/modules/nginx-echo 
--add-module=/usr/src/nginx/source/nginx-1.2.4/debian/modules/nginx-upstream-fai
r --add-module=/usr/src/nginx/source/nginx-1.2.4/debian/modules/nginx-syslog 
--add-module=/usr/src/nginx/source/nginx-1.2.4/debian/modules/nginx-cache-purge 
--add-module=/usr/src/nginx/source/nginx-1.2.4/debian/modules/naxsi/naxsi_src

Please provide your nginx configuration any additional information below.
########################################################
/etc/nginx/nginx.conf:
user application;
worker_processes 8;
pid /var/run/nginx.pid;

events {
    worker_connections 768;
    # multi_accept on;
}

http {
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;

include /etc/nginx/mime.types;
default_type application/octet-stream;

##
# Logging Settings
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

##
# Gzip Settings
gzip on;
gzip_disable "msie6";
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/x-javascript 
text/xml application/xml application/xml+rss text/javascript;

include /etc/nginx/naxsi_core.rules;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
#########################################
/etc/nginx/sites-enabled/site
server {
      listen   10.0.0.4:443 ssl;
      server_name example.com;
      root /var/www/example.com/;
      proxy_pass_header Server;

access_log /var/log/nginx/example_access.log;
error_log  /var/log/nginx/example_error.log error;

ssl_protocols SSLv3 TLSv1;
ssl_certificate      /etc/nginx/example.com.crt;
ssl_certificate_key  /etc/nginx/example.com.key;
ssl_ecdh_curve secp521r1;

sendfile off;
send_timeout 360;
proxy_set_header   X-Forwarded-Proto https;
proxy_set_header   Host             $host;
proxy_set_header   X-Real-IP        $remote_addr;
proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
proxy_set_header   X-Url-Scheme $scheme;
proxy_redirect     off;

location ~* ^/(images|javascripts|stylesheets|assets)/ {
 expires max;
 add_header Cache-Control public;
 add_header Last-Modified "";
 add_header ETag "";
 break;
}

location / {
 include /etc/nginx/naxsi_my.rules;
 try_files $uri @upstream;
}

location @upstream {
 proxy_pass  http://backend;
}

location /RequestDenied {
 return 418;
}
}

upstream backend {
  server x.x.x.x;
}
#################################################
/etc/nginx/naxsi_my.rules;
LearningMode;
SecRulesEnabled;
#SecRulesDisabled;
DeniedUrl "/RequestDenied";

include "/etc/nginx/naxsi_my.rules.d/my.rules";

## check rules
CheckRule "$SQL >= 8" BLOCK;
CheckRule "$RFI >= 8" BLOCK;
CheckRule "$TRAVERSAL >= 4" BLOCK;
CheckRule "$EVADE >= 4" BLOCK;
CheckRule "$XSS >= 8" BLOCK;

#################################################
/etc/nginx/naxsi_my.rules.d/my.rules
BasicRule wl:1005,1309 "mz:$HEADERS_VAR:cookie";
BasicRule wl:1009 "mz:$BODY_VAR:param2";
BasicRule wl:0 "mz:$BODY_VAR:user%5bparam%5d";

Original issue reported on code.google.com by ja.luc...@zhr.pl on 16 Oct 2012 at 1:48

Attachments:

• core_dump_backtrace.txt
• nginx_debug.log.txt

[GoogleCodeExporter]

Hello !

Thank you very much for the report, we will investigate quickly.
Unfortunately, I was not able to reproduce it with a quick testing session, 
hope I will have some spare time tonight to play with it ;)

Could you please try to remove naxsi from configuration and see if it happens 
again ?
(As I don't see any reference to naxsi in the stacktrace, I would dare to 
suspect nginx itself)

If you are around, could you join #naxsi on freenode so we can discuss the 
potential issue ?

Best regards,

Original comment by ori...@gmail.com on 16 Oct 2012 at 2:02

• Added labels: Priority-Critical
• Removed labels: Priority-Medium

[GoogleCodeExporter]

When I turned off the naxsi (comment out one "include" in nginx.conf and 
sites-enabled/site) the error does not occur.
I have just checked another configuration: Naxsi on (learning mode), turned off 
the proxy to the backend application (nginx serving only a few static files), 
and also the problem does NOT occur (I've generated about 1k queries and see 
still no error).
So the problem seems to be related to naxsi + proxy connections. Maybe I'm 
wrong, but think that my configuration of reverse-proxy function is correct.. 
Although it is little complicated.

I do not use irc and I'm not members of freenode, but tomorrow or the day after 
tomorrow I'll try to find the time to change that.

Best regards,
Jack

Original comment by ja.luc...@zhr.pl on 16 Oct 2012 at 2:44

[GoogleCodeExporter]

Thanks for that quick answer, I'll try to make some tries in this direction.

Nice finding by the way, don't know yet who's to blame : naxsi or nginx, maybe 
both ;)

Original comment by ori...@gmail.com on 16 Oct 2012 at 2:58

[GoogleCodeExporter]

I am experiencing the same problem. I am using nginx + naxsi in a reverse proxy 
configuration.
The problem occurs when naxsi's rules are being hit. If I disable naxsi or 
whitelist all the rules the problem goes away.

Original comment by carlos.l...@gmail.com on 24 Oct 2012 at 5:20

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Looks like I found a workaround.
The transfer of three "proxy_set_header" directives from global "server" 
section to the "location /" caused the errors no longer appear: 
proxy_set_header   X-Real-IP        $remote_addr;
proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
proxy_set_header   X-Url-Scheme $scheme;

Tested with 5k queries "blocked" by naxsi in LearningMode, and is OK. In normal 
(block) mode still allways work correctly too.

I have not found a reason for my original configuration would be flawed, but it 
seems that some of the variables in the header should be modified only for 
proxy section (section "location /" directing to @upstream) and should not be 
modified for local files. Perhaps it is more correct configuration, but it 
doesn't explain, why without naxsi it work good, and with naxsi caused "worker 
process # exited on signal 11".
Any idea to confirm this in a different way than the experimentally? Any 
sources about correct place for proxy_set_header directives? 
On the internet (http://lmgtfy.com/?q=proxy_set_header+nginx+signal+11) I saw 
that on BSD with other modules (not naxsi) people have also had a problem with 
"exited on signal 11" with "proxy_set_header" in "location", so I'm afraid that 
I have not found a solution, but only a workaround.

Below part of my current configuration.

Best regards,
Jack

#########################
/etc/nginx/sites-enabled/site
server {
 (..)
 proxy_set_header   X-Forwarded-Proto https;
 proxy_set_header   Host             $host;
 proxy_redirect     off;
#proxy_set_header   X-Real-IP        $remote_addr;
#proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
#proxy_set_header   X-Url-Scheme $scheme;

 location / {
   include /etc/nginx/naxsi_my.rules;
   proxy_set_header   X-Real-IP        $remote_addr;
   proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
   proxy_set_header   X-Url-Scheme $scheme;
   try_files $uri @upstream;
   }

 (..)
}
######################################

Original comment by ja.luc...@zhr.pl on 25 Oct 2012 at 9:12

[GoogleCodeExporter]

Hello,

I've just commited a patch that should fix this issue.
Could you try it and tell us if everything is working fine ?

Original comment by sephirot...@gmail.com on 26 Oct 2012 at 9:17

[GoogleCodeExporter]

It seems to have solved the problem. I've been testing for several hours now 
and not a single crash :)

Thank you!

Best regards,
Carlos

Original comment by carlos.l...@gmail.com on 26 Oct 2012 at 4:45

[GoogleCodeExporter]

Original comment by sephirot...@gmail.com on 26 Oct 2012 at 4:48

• Changed state: Fixed