nginx -t very slow with modsec enabled 15 sec !

[albgen]

Describe the bug

When the mod_sec module is enabled, the command nginx -t is very slow. I have something like 15 websites running on this webserver.

With modsec enabled and loaded

[root@web nginx]# time nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful

real 0m13.484s user 0m11.070s sys 0m1.334s

Without the modsec

[root@web nginx]# time nginx -t nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful

real 0m0.065s user 0m0.023s sys 0m0.027s

Rule Set (please complete the following information):

• Running OWASP
• What is the version number? latest

Current nginx version is 1.19.1

[zimmerle]

Hi @albgen,

Are you using any external resources? such as SecRemoteRules? What happens to be the ModSecurity version that you have running?

[albgen]

hi zimmerle,

I have created only one rule that i use just to understand if the waf is enabled or not in a web application and is the following:

Basic test rule SecRule ARGS:testparam "@contains test" "id:1234,deny,status:403"

All the rest of the rules is the owasp-modsecurity-crs. ModSec is compiled from git and is the latest version.

I have the same rules and same modsec compiled(the same way) and installed on another machine, and there i don't have this issue. The difference between the 2 installations is the nginx config files. So, what i think it could be the issue is that on the machine where the issue is present i define for each website one config file on the folder /etc/nginx/conf.d Also the one that has issues, is a web server with php...etc. The one where i don't have the issue is only acting as reverse proxy and nothing else.

hope to have given a clue on what could be the issue. I have also some memory leak problems on the same installation and i suspect is the modsec module. This is a very serious problem because it is eating all the server memory and it stops nginx. Currently i have disabled the modsec module(and all the other custom modules of nginx) just to understand if the memory leak will come out. If it does, it means that there is a problem on nginx. If the problem does not appear than the issue is on modsec. Here in the nginx mailing list i have posted the issue

[root@web ~]# nginx -V nginx version: nginx/1.19.1 built by gcc 4.8.5 20150623 (Red Hat 4.8.5-39) (GCC) built with OpenSSL 1.0.2k-fips 26 Jan 2017 TLS SNI support enabled configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib64/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' --with-ld-opt='-Wl,-z,relro -Wl,-z,now -pie'

[albgen]

Disabling the modsec module, the memory leakage disappeared and also as mentioned before the ngin -t command is very fast.

Any help please or thoughts?

Thanks

[AnoopAlias]

From what I have seen, nginx -t or time for restart/reload of Nginx is proportional to the number of virtual hosts you have. In the first setup and second setup , apart from having the config is separate files, could you confirm the number of the virtual hosts ( individual server {} ) are the same?

[albgen]

There are 16 config file. Each has at least 2 server directives and 2 of the config files has 4 server directive.

[AnoopAlias]

There are 16 config file. Each has at least 2 server directives and 2 of the config files has 4 server directive.

What I am asking is, in the 2 installations that you mentioned, are the number of server {} directives the same or do they differ?

[albgen]

Yes, they differ. On the other installation which is acting as reverse proxy, there are only 4 server directives...

[AnoopAlias]

ok as mentioned the load time is proportional to the number of server{} blocks. If you have like 100 or more it almost makes it impossible to use the modsec module( or other waf) from what I have seen

In fact, the case is similar with another WAF that I tested too, you can see the details here https://forum.nemesida-security.com/threads/regarding-nwaf-free-module.27/

My guess is that nginx somehow try to load all the rules into memory not once, but once per every server {} block, making it a load a lot of duplicates and thus exponentially increasing the time ( I may be wrong though )

[albgen]

The strange thing is that nobody has encountered this issue. Most probably, people does not use waf and websites on the same nginx instance

[AnoopAlias]

yes, it is strange. Often the server {} blocks are necessary for TLS. I am also wondering how CloudFlare would work with the WAF as they would have millions (or possibly more) of server {} blocks defined or they just have a few vhost with a single multi-domain TLS certificate?

It would be also helpful for someone who knows the Nginx internals to shed light on exactly what's going wrong in loading a WAF ( multiple ones like ModSec v3 and Nemesida are plagued by this issue)

[albgen]

Most probably cloudflare use some kind of containerization where each customer gets its own instance of waf container...who knows

[void-in]

Just to chip in with my own config, we have more than 100 vhosts with ModSecurity and it is working fine. The idea is not to load the rules in each server block separately but rather once. Then you can load your own custom rule files only in the server configs.

[albgen]

Just to chip in with my own config, we have more than 100 vhosts with ModSecurity and it is working fine. The idea is not to load the rules in each server block separately but rather once. Then you can load your own custom rule files only in the server configs.

Ok but how do you manage the need to enable/disable modsec on a particular vhost?

[albgen]

i moved the the directive that loads the rules on the nginx config file (modsecurity_rules_file /etc/nginx/modsec/main.conf;) On each server directive, than i decide if i have to enable or disable the modsec. In this way, it is very fast.

[snuffy1987]

Do you have status engine turned on? Are you able to resolve status.modsec.org from that host? Can you check with tcpdump port 53? Such behavior I encountered when I turned on modsec in an offline environment and it waited few seconds for response to status engine. It was not reachable of course. When I turned off status engine it worked fast again, because mod_sec was not resolving.

[zimmerle]

Closing this as I did not managed to reproduce it any longer on v3/master.

[albgen]

If you enable modsec module on each server directive, you will get the exact slow behaviour. Currentlt, to avoid the issue, I have enabled it on the main nginx.conf file

[zimmerle]

If you enable modsec module on each server directive, you will get the exact slow behaviour. Currentlt, to avoid the issue, I have enabled it on the main nginx.conf file

@albgen, The slowness to load the rules was reduced. If the load time is "N" and you load "Z" amount of times, the total time spent on loading will be in the order of magnitude N*Z; Not to mention that the memory utilization will also be a factor of "Z". It could be especially slow if you are using external resources that demand download from an external server.

The recommendation is to load globally and add exceptions locally.

[ministry-of-code]

I can second @albgen 's Problems. We have quite a complex Nginx configuration for a customer with many vhosts and we embed mod_security and nignx -t takes like 30-60 seconds until it completes. Also, there seems to be a memory leak. When we restart the Nginx with nginx -s reload it gets significant bigger over time until it gets killed from the Linux kernel because it consumes too much memory. We worked around the problem with stopping and starting the nginx every time from scratch.