Requests stuck in certain cases

FlorinAsavoaie commented 8 years ago

I raised nginx bug 801 for this but actually it seems to be an issue with Naxsi.

Long story short, here's the comment from the nginx developers:

"[...] it looks like a 3rd party module have read a request body in the rewrite phase (not something normally done, the body is expected to be read only at content phase), and failed to properly restore request handlers after this. That is, it's a naxsi bug."

buixor commented 8 years ago

Hi,

Thanks for the bug-report. In the configuration you pasted in your nginx ticket, I do not see any naxsi configuration, did you strip it on purpose, or does it means that it happens even if naxsi's code is not explicitly enabled ?

regards,

FlorinAsavoaie commented 8 years ago

I just stripped it but it happens if I enable it for the location only. If naxsi is not enabled for the location but just built into nginx, the bug does not occur. However, it doesn't matter if it is in learning mode or not.

buixor commented 8 years ago

Can you please provide a full non strippped config so I can reproduce it ?

FlorinAsavoaie commented 8 years ago

location /myapp {
  # Naxsi config
  LearningMode;
  SecRulesEnabled;

  DeniedUrl "/RequestDenied";

  CheckRule "$SQL >= 8" BLOCK;
  CheckRule "$RFI >= 8" BLOCK;
  CheckRule "$TRAVERSAL >= 4" BLOCK;
  CheckRule "$EVADE >= 4" BLOCK;
  CheckRule "$XSS >= 8" BLOCK;

  auth_request /auth;
  proxy_pass http://some-upstream;
}

location = /auth {
  proxy_method GET;
  proxy_pass_request_body off;
  proxy_pass_request_headers off;
  if ($cookie_auth) {
    set $app_auth "Token $cookie_auth";
  }
  if ($http_authorize) {
    set $app_auth $http_authorize;
  }
  proxy_set_header Authorize $app_auth;
  proxy_set_header Host $host;
  proxy_set_header Content-Length "";
  proxy_cache proxyCacheAuth;
  proxy_cache_key $remote_addr$host$app_auth;
  proxy_cache_valid any 15m;
  proxy_pass http://authupstream/random/url;
}

buixor commented 8 years ago

thanks for the details, I will look into it

buixor commented 8 years ago

Hello,

Sorry, but I did not manage to reproduce the issue, can you give me a hand ? (I use naxsi dev's env, so we have something reproducible)

cp -R .../nginx/nginx-1.9.1 /tmp/
cd naxsi_src
make re deploy

(note: this deploys a dummy nginx in /tmp/naxsi_ut)

Then I edit nginx.conf to make it look like yours (/tmp/naxsi_ut/nginx.conf)

server {
  listen 4242;
  server_name  localhost;

location /myapp {
  # Naxsi config
  LearningMode;
  SecRulesEnabled;

  DeniedUrl "/RequestDenied";

  CheckRule "$SQL >= 8" BLOCK;
  CheckRule "$RFI >= 8" BLOCK;
  CheckRule "$TRAVERSAL >= 4" BLOCK;
  CheckRule "$EVADE >= 4" BLOCK;
  CheckRule "$XSS >= 8" BLOCK;

  auth_request /auth;
  proxy_pass http://77.153.128.117/;
  proxy_set_header Host www.google.fr;
#  root html;
}
location = /auth {
  proxy_method GET;
  proxy_pass_request_body off;
  proxy_pass_request_headers off;
  if ($cookie_auth) {
    set $app_auth "Token $cookie_auth";
  }
  if ($http_authorize) {
    set $app_auth $http_authorize;
  }
  proxy_set_header Authorize $app_auth;
  proxy_set_header Host $host;
  proxy_set_header Content-Length "";
  proxy_pass http://localhost:4241/random/url;
}

(localhost:4241 is just a netcat so I can answer 200 OK by hand)

GET request : I get the answer without problem, as you described.

POST request :

$ curl -d "test" http://127.0.0.1:4242/myapp
<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 405 (Method Not Allowed)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}ht
....

(the scrapped HTML is google's answer)

in the meanwhile in the netcat :

$ nc  -vlp 4241
listening on [any] 4241 ...
connect to [127.0.0.1] from localhost [127.0.0.1] 60381
GET /random/url HTTP/1.0
Host: localhost
Connection: close

HTTP/1.1 200 OK

And I get the answer here.

Please let me know what I missed to reproduce your issue !

Regards,

andreidotnet commented 8 years ago

Hello, I will provide soon a full configuration that is confirmed from our side to have this issue. This has not been abandoned from our side :)

andreidotnet commented 8 years ago

After investigating further we've tracked the bug to the size of the request, it seems to be working until up to 1024.

The following should not work with your config : dd if=/dev/zero bs=1 count=1024 | curl -vvk 'http://127.0.0.1:4242/myapp' -X PUT --data-binary @-

The following should work, with your config : dd if=/dev/zero bs=1 count=10 | curl -vvk 'http://127.0.0.1:4242/myapp' -X PUT --data-binary @-

Note that if you comment #SecRulesEnabled; and retry the request with a content length greater than 1024, it will work.

Please let us know if you were able to reproduce the issue with the settings you have posted above.

buixor commented 8 years ago

thank you, I'm trying to give it a look soon :)

andreidotnet commented 8 years ago

Just noticed, we're using naxsi-0.54rc3. Will test the latest release soon and confirm if we see the same problem there.

buixor commented 8 years ago

Sadly, I'm fraid the latest won't fix the issue, none of the mechanics where changed :/

On Fri, Oct 23, 2015 at 10:07 AM, Andrei S. notifications@github.com wrote:

Just noticed, we're using naxsi-0.54rc3. Will test the latest release soon and confirm if we see the same problem there.

— Reply to this email directly or view it on GitHub https://github.com/nbs-system/naxsi/issues/226#issuecomment-150502675.

andreidotnet commented 8 years ago

So we can take this as a confirmation that you were able to reproduce the issue ? :)

buixor commented 8 years ago

Not yet, but you seem to have provided some quite precise things for me to do so. I'll try not to be too lazy this week-end and give it a more serious look ;)

On Fri, Oct 23, 2015 at 10:28 AM, Andrei S. notifications@github.com wrote:

So we can take this as a confirmation that you were able to reproduce the issue ? :)

— Reply to this email directly or view it on GitHub https://github.com/nbs-system/naxsi/issues/226#issuecomment-150511075.

buixor commented 8 years ago

Sorry, I failed you and had a lazy week-end :p But I'll check it asap, just that it might requires some serious dive-in :)

FlorinAsavoaie commented 8 years ago

Yeah, it probably will. I wasn't sure if it is really a pretty nasty problem or it is just me not having enough skills :)...

buixor commented 8 years ago

Hi,

Ok, just tested it :)

At least (good for you, bad for me), I can confirm the bug exists.

buixor commented 8 years ago

Hello,

I didn't gave up, but I still didn't understand what is happening, as it's something I never see. I guess I will have to turn myself to the developper's ML for help :)

andreidotnet commented 8 years ago

Hello, Any luck so far with the discovered issue ?

buixor commented 8 years ago

Hi,

I worked a bit on it this week-end and I'm getting closer to it I guess, but I don't have yet a fix to submit, I still need to play around a bit !

I didn't give up and I'll keep you posted as soon as I have something that seems plausible :)

buixor commented 8 years ago

Hi & happy new year !

Some further testing have provided some interesting things (no, i didn't solve the issue yet):

form-input-nginx-module (by agentzh) is often cited as a reference (on nginx-dev ML) for modules reading the request body in the REWRITE phase.
if I produce a setup combining form-input-nginx-module + mod_auth without naxsi, I still get the bug (based on the provided test-case)

So far, I think that either nginx changed something in the internals (since nginx 1.7.4), or I've been wrong the whole time, or mod_auth as an issue. Just to be sure I'm not brain-fried, can you confirm that the following setup still gives you the same issue

cd .../naxsi_src/
git clone http://github.com/simpl/ngx_devel_kit.git
git clone http://github.com/calio/form-input-nginx-module.git

Remove naxsi from ./configure, add form-input :

 --add-module=$(MOD_PATH)/ngx_devel_kit \
 --add-module=$(MOD_PATH)/form-input-nginx-module \
 --with-http_auth_request_module \
 --with-debug

I used the following configuration :

location /myapp {
# ensure client_max_body_size == client_body_buffer_size
client_max_body_size 100k;
client_body_buffer_size 100k;
set_form_input $data;    # read "data" field into $data
set_form_input $foo foo; # read "foo" field into $foo
auth_request /auth;
proxy_pass http://77.153.128.117/;
proxy_set_header Host www.google.fr;
}

Cheers, and thanks for your patience !

ps: that's a really sneaky tricky *\ bug :)

FlorinAsavoaie commented 8 years ago

I rather think that both naxsi and the other module do something that they are not supposed to do in the wrong phase. It is possible that this changed at some point in nginx but it doesn't change the fact that the nginx devs do not recommend doing that. We will give it a try as soon as we can and confirm.

buixor commented 8 years ago

Hi !

I didn't mean that mod_auth is wrong, I definitely trust maxim on this kind of topic ^^

If you confirm the issue, I'm going to turn to nginx-dev's ml.

Using form-input-nginx-module as a use case is definitely easier than naxsi, as the code is way shorter and cleaner !

cheers,

andreidotnet commented 8 years ago

Hello, Sorry for the late reply, got caught up with other stuff. Tested today with nginx 1.9.9 with the requested configuration mentioned previously.

/nginx/usr/sbin/nginx -V nginx version: nginx/1.9.9 built by gcc 4.4.7 20120313 (Red Hat 4.4.7-16) (GCC) built with OpenSSL 1.0.1e-fips 11 Feb 2013 TLS SNI support enabled configure arguments: --add-module=../ngx_devel_kit --add-module=../form-input-nginx-module --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --user=nginx --group=nginx --with-http_ssl_module --with-http_realip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_auth_request_module --with-stream --with-file-aio --with-http_v2_module --without-http_fastcgi_module --without-http_scgi_module --without-http_uwsgi_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic'

The nginx config

location /myapp {

ensure client_max_body_size == client_body_buffer_size

client_max_body_size 100k; client_body_buffer_size 100k; set_form_input $data; # read "data" field into $data set_form_input $foo foo; # read "foo" field into $foo

auth_request /auth; proxy_pass http://77.153.128.117/; proxy_set_header Host www.google.fr; }

With

dd if=/dev/zero bs=1 count=10 | curl -vvk 'http://127.0.0.1:4242/myapp' -X PUT --data-binary @-

we get an ok response

With

dd if=/dev/zero bs=1 count=1024 | curl -vvk 'http://127.0.0.1:4242/myapp' -X PUT --data-binary @-

the request gets stuck

1024+0 records in 1024+0 records out 1024 bytes (1.0 kB) copied, 0.00100387 s, 1.0 MB/s

About to connect() to 127.0.0.1 port 4242 (#0)

Trying 127.0.0.1... connected

Connected to 127.0.0.1 (127.0.0.1) port 4242 (#0)

PUT /myapp HTTP/1.1 User-Agent: curl/7.19.7 (x8664-redhat-linux-gnu) libcurl/7.19.7 NSS/3.19.1 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2 Host: 127.0.0.1:4242 Accept: /_ Content-Length: 1024 Content-Type: application/x-www-form-urlencoded

^C

When i get some time i'll try and test it out with previous nginx version to try and trace down the last version this worked ok .. if that helps.

buixor commented 8 years ago

I'm going to shoot an email to nginx-dev's ML to check. I'll mostly ask if form-input-nginx-module is doing things correctly :)

Keep me posted as well,

cheers !

andreidotnet commented 8 years ago

Did some more testing with multiple nginx version. We can reproduce the issue describe in the previous post even in nginx 1.5.4 (first version with http_auth_request_module Without http_auth_request_module, the request works as expected even in the latest nginx builds. It's starting to look like http_auth_request_module might be the issue.

buixor commented 8 years ago

Hi,

I posted on nginx-dev ML : http://mailman.nginx.org/pipermail/nginx-devel/2016-January/007762.html So far, I didn't get the perfect answer but some good clues. Maxim suggested something I will need to try (first attempt failed), while agentzh tends to point at auth module ;)

I didn't give up ;)

andreidotnet commented 8 years ago

Heya buixor, Did you manage to find any news regarding this issue ? Have not noticed any activity on the ML regarding this topic.

buixor commented 8 years ago

Hello,

Sorry for delay :) I tried Maxim suggestion, but without success. Tbh I think agentzh might be right, as I didn't manage to reproduce the bug with any other module except this one. I will try to get back on it after we release 0.55, I didn't had much chance to work on this issue :(

FlorinAsavoaie commented 7 years ago

Hi, how are you doing?

0.55 is out 💃 Time for party!

But we'd also appreciate some more input on this one.

Thanks,

defanator commented 5 years ago

Any success with this one so far?

FlorinAsavoaie commented 5 years ago

We switched to mod_security :(

jvoisin commented 5 years ago

Aw </3

defanator commented 5 years ago

Well, don't be surprised if you'll see the same behavior with (lib)modsecurity. :)

https://github.com/SpiderLabs/ModSecurity-nginx/issues/130 https://github.com/SpiderLabs/ModSecurity-nginx/pull/131

dani commented 5 years ago

Guess I have somehow the same, or a similar/related issue. Running nginx 1.15.6 and naxsi 0.56. When naxsi is enabled, even in learning mode, some POST requests just hang. I haven't isolated exactly which ones, but apparently, it can affect requests < 1024. I'll try to get some more info and a simple reproducer

dani commented 5 years ago

For now, I can confirm the issue is only present when I protect the location with auth_request. I'm using Lemonldap::NG handler (see configuration example here: https://lemonldap-ng.org/documentation/latest/configvhost#reverse_proxy1)

As soon as I remove the auth_request part, everything is working again.

buixor commented 5 years ago

@FlorinAsavoaie Can you please notify us if you encounter the same issue with mod_sec and auth_request ? If it's the case, I should document the known incompatibility between those two modules.

dani commented 5 years ago

Has something like https://github.com/SpiderLabs/ModSecurity-nginx/pull/131/commits/1cd54842a58485fc4b50a7a4fdb1045fc2a4a844 been applied on NAXSI ?

buixor commented 5 years ago

@dani This is what (I think but ofc I didn't keep proper track of it) I tried after maxim's suggestion when I posted, but it's worth giving a try again :)

dani commented 5 years ago

I'd be happy to give it a try, if you can show me a commit. I have a setup where I can easily reproduce the issue.

buixor commented 5 years ago

@dani in our case, the changes should be made in ngx_http_dummy_access_handler of naxsi_skeleton.c around this part.

Unfortunately, it seems I didn't keep track of my previous attempts (no commit), so I guess I'll have to first spend some time to do the restoration of event_handlers in a branch for you to properly test :/

nbs-system / naxsi

Requests stuck in certain cases #226

ensure client_max_body_size == client_body_buffer_size