Wordpress configuration and speed issues under load test...

[thewzrd]

Forgive my ignorance if this has been answered elsewhere or if this just super feeble minded on my part but I'm hitting a snag I can't seem to sort out.

I'm doing load testing (from loader.io) and performance is just getting crushed with the waf active.

With everything I had read (sub 1 ms impact) I assumed that this wouldn't happen. So I believe that I've gotta be messing up the configuration somewhere along the line.

Is there a way for the waf to trigger AFTER the cache such that if you're just running a simple load test at the home page the waf won't even come into the picture?

Thanks for all your hard work on this project... I'd love to bring it permanently into our production stack. I'd be happy to contribute some documentation for uber beginners, once I move slightly out of that category.

[p0pr0ck5]

Hi,

Can you post your full configuration, along with the output of nginx -V? It will be helpful to know what rulesets and regex engine options you're running with, as well as examining what the performance drop looks like. Without knowing your full config (both lua-resty-waf and nginx config), there's no way to know how cache or WAF performance is making a difference. :) The more detail the better!

[thewzrd]

Thanks for getting back to me. I think I've almost got it cracked. I'll post back either way - if I get to the other side or if I need to include all the various details of what I'm running and where things are breaking.

Long story short: I was doing it wrong. Fingers crossed that I'm not anymore.

[p0pr0ck5]

Thanks for the update! Yes, regardless of the outcome, let us know- perhaps someone can learn from your usage, or maybe you've stumbled upon a magic bug ;)

[thewzrd]

I've definitely gotten the errors way down. I was in the 25% range (I'm doing pretty hefty 0 to 5000 concurrent users tests on a LEMP WP stack).

Now we're down to around 1.5% errors with the firewall effectively off and around 7% with it on. What I'm not really sure about and I can't seem to find anything to conclusively "wins" is where to put these commands:

access_by_lua_block {
    local lrw = require "resty.waf"
    local waf = lrw:new()
    waf:set_option("debug", true)
    waf:set_option("mode", "ACTIVE")
waf:set_option("event_log_target", "file")
waf:set_option("event_log_target_path", "/tmp/waf.log")
waf:set_option("event_log_request_headers", true)
waf:set_option("event_log_request_arguments", true)
waf:set_option("event_log_request_body", true)
waf:set_option("event_log_periodic_flush", 1)
waf:exec()
}

log_by_lua_block {
    local lrw = require "resty.waf"
    local waf = lrw:new()
    waf:write_log_events()
}

I pulled those from the Dreamhost "How To Install OpenResty and Lua-WAF" walkthrough.

Should they go inside both location / and location .php seen below...

location / {
      try_files $uri $uri/ /index.php?$args;
}
location ~ \.php$ {
      fastcgi_index index.php;
      fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;  
      try_files $uri =404;
      include fastcgi_params;
      fastcgi_pass 127.0.0.1:9070; 
      fastcgi_cache_bypass $skip_cache;
      fastcgi_no_cache $skip_cache;
      fastcgi_cache WORDPRESS;
      fastcgi_cache_valid  60m;
}

As you can see I'm using nginx fastcgi caching which is normally blazing, but it seems like wherever I embed the firewall it's just getting pounded. I realize this is a pretty insane load (5K concurrent users) but I'm sure that Cloudflare is handling many many multiples of that. And so I'm sure that I'm still doing it all wrong here.

Just not nearly as wrong as I was doing it before.

Totally sort of unrelated sidenote: how difficult would it be for me to adapt a rule that I was using with Comodo and ModSec to do live scanning of file uploads through ClamAV? I've had no luck translating their massive collection of rules over but I'd like to be able to knock out that one in particular. Is there any kind of simple rule to where I can pass uploads through the clamav hook?

THANK YOU! for everything. I'll likely be writing some kind of walkthrough once I'm done here. It'd be wasteful just for my own future reference to not have some of this process recorded.

[p0pr0ck5]

Hi,

Quick note, I updated your comment for some basic syntax highlighting for readability :)

First, turn debug mode off (just remove waf:set_option("debug", true), it's false by default). This the biggest contributor to slowdown (I need to make a note in that DreamHost walkthrough to remove that once initial testing is done).

Next, what's the output of nginx -V? If you don't have OpenResty built with PCRE JIT, you're going to notice a huge slowdown when running with lots of rules that use regex (even if the expression isn't particularly complex, it's still a waste of cycles to compile every time). See https://www.cryptobells.com/building-openresty-with-pcre-jit/.

The more (and complicated) rules you run, the longer execution will take. Considering disabling any provided rulesets that you don't need (XSS and SQLi are pretty expensive; they're pulled from the OWASP CRS v2 ruleset, and are in pretty bad need of an update- it's on my to-do ;) ). As for fCGI caching, that's going to be done after access phase processing is done, so fCGI-cached responses will still be inspected by lua-resty-waf here. Putting a caching layer in front of the WAF isn't terribly tricky, but requires a bit more engineering work than I can afford to walk you through here :) And yes, Cloudflare can handle a lot of requests- that's what happens when you dump millions of dollars into infrastructure and engineering ;)

As for a ClamAV hook- well, there's no such thing as a "hook" for clam ;) In ModSecurity, that's implemented by executing a separate process (typically a Perl script or some such that the rule author defines). I haven't bothered with translating the exec ModSecurity action- for something that needs that level of complexity, you'd be better off integrating that work directly inside an OpenResty handler, rather than trying to shove it inside lua-resty-waf. Interestingly enough, I have been working on a project to provide bindings to the Yara malware scanning library for OpenResty. There are no plans to integrate that directly into lua-resty-waf, but that may be something worth looking at. I'll post a link here once that project is available.

[thewzrd]

So first things first: you're awesome. This is insanely good support in exchange for... nothing but more wooly questions from my end :) You need a Buy Me A (Case of) Beer button somewhere because I would hammer that.

Second: Nginx -v output

nginx version: openresty/1.11.2.2 built by gcc 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) built with OpenSSL 1.0.2j 26 Sep 2016 TLS SNI support enabled configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt=-O2 --add-module=../ngx_devel_kit-0.3.0 --add-module=../echo-nginx-module-0.60 --add-module=../xss-nginx-module-0.05 --add-module=../ngx_coolkit-0.2rc3 --add-module=../set-misc-nginx-module-0.31 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.06 --add-module=../srcache-nginx-module-0.31 --add-module=../ngx_lua-0.10.7 --add-module=../ngx_lua_upstream-0.06 --add-module=../headers-more-nginx-module-0.32 --add-module=../array-var-nginx-module-0.05 --add-module=../memc-nginx-module-0.17 --add-module=../redis2-nginx-module-0.13 --add-module=../redis-nginx-module-0.3.7 --add-module=../rds-json-nginx-module-0.14 --add-module=../rds-csv-nginx-module-0.07 --with-ld-opt=-Wl,-rpath,/usr/local/openresty/luajit/lib --add-module=/usr/local/src/openresty-1.11.2.2/../ngx_cache_purge-2.3 --with-pcre=/usr/local/src/pcre-8.40 --with-http_v2_module --with-pcre-jit --with-http_ssl_module

Third: I can't believe I missed that Debug line. That would definitely make a big difference. Unfortunately, now that I've pulled that out... the errors have gone up... ? I was a shade under 5% and now I'm back up in the 17% range. It seems like with that off, the system is detecting it as a SYN flood and just dropping everything. With Debug on that wasn't happening. So now I'm trying to unpack that problem.

I'm also going to try disabling those rules you mentioned and see what happens with a little more testing.

The thing that is really impressive to me is that with me doing this WRONG (Debug on etc) you're still performing really really well. This same scale of load test thrown at a crazy expensive WPEngine Enterprise account errors out like 30% of the time. And I'm sending it at a $20/month Vultr box.

Finally: clearly a good chunk of this is way over my head but I get enough to understand the basic context of what you're talking about. I appreciate you taking the time to watch me learn to crawl :) Maybe one day soon I'll be walking. Yeah, the "hook" I was talking about was what they called the perl file from the walkthrough that I followed for the ModSec implementation. As to the "caching in front of the WAF" can you point me towards anything that you think would help me hack through that? I had previously thought of doing all of this as: Nginx caching in front of Apache running ModSec. But I'd like to keep it all in Nginx. As to your point about doing the clamAV scanning inside of an OpenResty handler... that one sounds well beyond my capability. Though it won't stop me from doing a little digging.

Thank you again for all your time.

One final note, because I wasn't 100% sure of your recommendation: I've got the WAF code dropped in inside the location PHP block only right now. And with Debug on, that's when I got the ~ 4.5% errors - the best results yet. I'm guessing that's the best place to keep it... I don't need to have it in both location blocks, do I?

Again, I apologize for my ignorance in all of this. And I really appreciate your time. I'd be happy to send you those beers, or coffee, or some other woefully insufficient measure of thanks.

[p0pr0ck5]

So first things first: you're awesome. This is insanely good support in exchange for... nothing but more wooly questions from my end :) You need a Buy Me A (Case of) Beer button somewhere because I would hammer that.

Woop! Tell your friends! (And if you're in the LA area, I'll totally let you buy me a beer ;) ).

Second: Nginx -v output

Ok, modern PCRE with JIT. Great!

Third: I can't believe I missed that Debug line. That would definitely make a big difference. Unfortunately, now that I've pulled that out... the errors have gone up... ? I was a shade under 5% and now I'm back up in the 17% range. It seems like with that off, the system is detecting it as a SYN flood and just dropping everything. With Debug on that wasn't happening. So now I'm trying to unpack that problem.

This sounds like the server is responding fast enough that the stress tester is sending more TCP handshakes than the server handle. This is almost certainly unrelated to the WAF, as this is happening further down the stack- essentially, you're overloading your box with TCP handshake traffic. Make sure you (and your client) has keepalive turned on. Again, unrelated to WAF, it might be worth having a look at tuning the network portion of your box: http://www.nateware.com/linux-network-tuning-for-2013.html#.WJO-4twrJhE (or google "tuning Linux networking performance" or some such ;) ).

The thing that is really impressive to me is that with me doing this WRONG (Debug on etc) you're still performing really really well. This same scale of load test thrown at a crazy expensive WPEngine Enterprise account errors out like 30% of the time. And I'm sending it at a $20/month Vultr box.

OpenResty lets you do some pretty great stuff :D

As to the "caching in front of the WAF" can you point me towards anything that you think would help me hack through that? I had previously thought of doing all of this as: Nginx caching in front of Apache running ModSec. But I'd like to keep it all in Nginx.

There are a number of ways to do this, with a little bit of work. You have have an Nginx server that handles client requests and caches data, and forwards data to an internal location/server; this would take a bit of engineering work and some familiarity with Nginx configs to get it working. You could also look at setting up Varnish as a cache in front of the OpenResty, if you expect to server mostly cached content (this is fairly straightforward, at the maintenance cost of running two daemons, but it's a fairly common design pattern). There are other ways to work on this too; feel free to join the community in either #openresty, #nginx, or #lua-resty-waf rooms on freenode (or all three!).

One final note, because I wasn't 100% sure of your recommendation: I've got the WAF code dropped in inside the location PHP block only right now. And with Debug on, that's when I got the ~ 4.5% errors - the best results yet. I'm guessing that's the best place to keep it... I don't need to have it in both location blocks, do I?

If you want to run it on every request, you could put the access_by_lua directive in the server {} block, outside the location {} block. If you only want to run it for requests handled by WordPress, and not static media (assuming you have Nginx configured appropriately), the PHP block would be an appropriate spot (note that lua-resty-waf has rules built in to not run checks against static media files, but handling this exemption through your config would be even more performant).

Again, I apologize for my ignorance in all of this. And I really appreciate your time. I'd be happy to send you those beers, or coffee, or some other woefully insufficient measure of thanks.

Happy to help! Tell your friends about us, keep asking questions, and thanks for your support! And if your ever at Nginx conference or SCaLE, let me know ;) And let me know if you have any other questions (feel free to also pop into the freenode rooms I mentioned above, I'm typically lurking in those and a few others like #modsecurity most of the time).

[p0pr0ck5]

@thewzrd also, check out lua-resty-core if you haven't as that will also improve performance noticeable. I have a write-up about this here: https://www.cryptobells.com/resty-core-the-good-the-bad-and-the-jit-y/

[p0pr0ck5]

@thewzrd I'm going to close this issue as there doesn't seem to be any more actionable work to be done here. If you have any more feedback or questions, please feel free to respond back or open a new issue!