search

mitchellkrogza / nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail for Repeat Offenders
Other
4.07k stars 484 forks source link

[INSTALLATION] directives seem to conflict with brotli, modsecurity, and pagespeed modules. Running sudo nginx -t fails. #462

Open Danrancan opened 2 years ago

Danrancan commented 2 years ago

Did you look through existing ISSUES ?

Yes.

Describe the problem you are experiencing

It seems that somehow my pagespeed, modsecurity, and brotli nginx modules are causing duplicate directives in the bad-bot-blocker configuration files, as my nginx test is showing non-existent duplicate directives. What can I do to figure this out? Please help?

Error Messages

When running the nginx test after installation, sudo nginx -treturns the following error:

nginx: [emerg] "variables_hash_max_size" directive is duplicate in /etc/nginx/conf.d/botblocker-nginx-settings.conf:18
nginx: configuration file /etc/nginx/nginx.conf test failed

Following that, I run a grep search for the duplicate directive...

sudo grep -r "variables_hash_max_size" /etc/nginx/*
/etc/nginx/conf.d/botblocker-nginx-settings.conf:variables_hash_max_size 4096;
Binary file /etc/nginx/modules/ngx_http_brotli_static_module.so matches
Binary file /etc/nginx/modules/ngx_http_modsecurity_module.so matches
Binary file /etc/nginx/modules/ngx_pagespeed.so matches

I have also checked all of my .conf and .vhost files manually for duplicate directives, none of which can be found.

Copy of nginx.conf


# Nginx conf file
user www-data;
worker_processes auto;
pid /run/nginx.pid;
#include /etc/nginx/modules-enabled/*.conf;
load_module modules/ngx_http_security_headers_module.so;
load_module modules/ngx_http_modsecurity_module.so;
load_module modules/ngx_pagespeed.so;
load_module modules/ngx_http_cache_purge_module.so;
load_module modules/ngx_http_brotli_filter_module.so;
load_module modules/ngx_http_brotli_static_module.so;

events {
    worker_connections 768;
    # multi_accept on;
}

http {
    ##
    # Security Headers
    ##

    security_headers on;
        hide_server_tokens on;

    ##
    # Mod Security
    ##

    modsecurity on;
    #modsecurity off;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;

    ##
    # Basic Settings
    ##

    # Upload settings
    client_max_body_size 200M;
    client_body_buffer_size 200M;

    sendfile on;
    tcp_nopush on;

    ##
    # Nginx Bad Bot Blocker Includes
    # REPO: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
    ##
    include /etc/nginx/conf.d/botblocker-nginx-settings.conf;
    include /etc/nginx/conf.d/globalblacklist.conf;

    tcp_nodelay on;
    types_hash_max_size 4096;
        server_tokens off;
        server_names_hash_bucket_size 64;
        server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

    ##
    # Brotli Compression Settings
    ##

    # The 'brotli off|on' value enables or disbles dynameic or on the fly compression of the content.
    brotli          on;
    # The 'brotli_static on' value enables the Nginx server to check if the pre-compressed files with the .br extensions
    # exist or not. The always value allows the server to send pre-compressed content without confirming if the browser
    # supports it or not. Since Brotli is resource-intensive, this modules is best suited to reduse the bottleneck situations.
    brotli_static       on;
    # The brotli_comp_level directive sets the dynamic compression quality. It can range from 0 to 11.
    brotli_comp_level   6;
        brotli_window       512k;
    # Configure a minimum length in order to have the requst compressed, determined by the Content-Length field in the HTTP headers.
        brotli_min_length       21; #20
    # Enable dynamic compression for specific MIME types, whereas text/html respnosese are always compressed.
    #brotli_types       application/atom+xml application/javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype application/x-font-ttf application/x-javascript application/xhtml+xml application/xml font/eot font/opentype font/otf font/truetype image/svg+xml image/vnd.microsoft.icon image/x-icon image/x-win-bitmap text/css text/javascript text/plain text/xml;
    brotli_types application/atom+xml application/javascript application/json application/rss+xml
             application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype
             application/x-font-ttf application/x-javascript application/xhtml+xml application/xml
             font/eot font/opentype font/otf font/truetype image/svg+xml image/vnd.microsoft.icon
             image/x-icon image/x-win-bitmap text/css text/javascript text/plain text/xml;

    ##
    # Gzip Compression Settings
    ##

    gzip off;
    # Linuxbabe turns gzip_vary on, but we turn it off here because when enabling it, you may have problems clearing the cache.
    #gzip_vary  off;
    #gzip_vary  on;
    #gzip_proxied   any;
    #gzip_min_length    1000;
    #gzip_comp_level    6;
    #gzip_buffers   16 8k;
    #gzip_http_version  1.1;
    # Linuxbabe (+ added extensions)
    #gzip_types application/json application/x-javascript application/javascript application/atom+xml application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml;

    ##
    # Proper Log Format for Netdata
    ##

    # Create a custom Nginx log format called netdata that includes information about request_time, and upstream_response_time, measured in seconds with millisecond resolution.
    log_format netdata '$remote_addr - $remote_user [$time_local] '
                  '"$request" $status $body_bytes_sent '
                  '$request_length $request_time $upstream_response_time '
                  '"$http_referer" "$http_user_agent"';

    ##
    # WebP Page Cache settings/conditional variables # Use this to enable SERVING webP images in nginx. Dont forget to add a location directive for webP in each virtual configuration files.
        ##

        #map $http_accept $webp_suffix {
          #default   "";
          #"~*webp"  ".webp";
        #}

    ##
    # SSL Settings
    ##

    ssl_session_cache   shared:SSL:10m; #SSL session cache
    ssl_session_timeout 1h;
    ssl_prefer_server_ciphers   on;

    ##
    # Logging Settings
    ##

    access_log  /var/log/nginx/access.log;
    error_log   /var/log/nginx/error.log;

    ##
    # FastCGI Cache (by Linuxbabe)
    ##

    # Use this cache path for caching to disk not ram
    #fastcgi_cache_path /usr/share/nginx/fastcgi_cache levels=1:2 keys_zone=phpcache:100m max_size=10g inactive=60m use_temp_path=off;
    # Use this cache path for caching to ram not disk
    fastcgi_cache_path      /usr/share/nginx/ramcache/fastcgi_temp/ levels=1:2 keys_zone=phpcache:100m max_size=2g inactive=60m use_temp_path=off;
    fastcgi_cache_key   "$scheme$request_method$host$request_uri";
    # By Easyengine.io
    # The line fastcgi_cache_use_stale is what makes caching on Nginx-side unique.
    # This line tells Nginx to use old (stale) cached version of page if PHP crashes. This is something not possible with WordPress caching plugins.
    fastcgi_cache_use_stale error timeout invalid_header http_500;
    fastcgi_read_timeout 300;
    # The next 3 lines work with cacheing directly to ram
    # https://websiteforstudents.com/improve-nginx-cache-performance-with-tmpfs-on-ubuntu/
    #fastcgi_cache_revalidate on;
    #fastcgi_cache_background_update on;
    #fastcgi_ignore_headers Cache-Control Expires Set-Cookie;

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

Copy of vhost / website / host .conf file


server {
  listen 80;
  listen [::]:80;
  server_name oddcake.net www.oddcake.net;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name oddcake.net  www.oddcake.net;
  root /var/www/oddcake.net/;

  ###
  # SSL (From Mozilla Config Generator: Modern Configuration)
  ###

  # Add Strict Transport Security Response Header with "always Paramater", to help prevent MITM attacks.
  # HSTS (ngx_http_headers_module is required) (63072000 seconds)
  add_header Strict-Transport-Security "max-age=63072000; includeSubDomains" always;
  ## Prevent click jacking by adding an X-Frame-Options header
  add_header x-frame-options "SAMEORIGIN" always;
  #add_header X-Frame-Options "SAMEORIGIN";
  # Add a content security policy header
  add_header Content-Security-Policy "frame-ancestors 'self';";
  # Secure MIME Types with X-Content-Type-Options. Below line adds the X-Frame-Options header in Nginx.
  add_header X-Content-Type-Options nosniff;
  # Enable X-XSS-Protection header in Nginx
  add_header X-XSS-Protection "1; mode=block";
  add_header Referrer-Policy "strict-origin";
  add_header Permissions-Policy "geolocation=(),midi=(),sync-xhr=(),microphone=(),camera=(),magnetometer=(),gyroscope=(),fullscreen=(self),payment=()";
  # User recommendation from linuxbabe for fastcgi to keep working
  #fastcgi_ignore_headers Cache-Control Expires Set-Cookie; # already in nginx.conf

  # Path to signed certificate
  ssl_certificate /etc/letsencrypt/live/oddcake.net/fullchain.pem; # Managed by admin
  # Path to Intermediate certificates
  ssl_certificate_key /etc/letsencrypt/live/oddcake.net/privkey.pem; # Managed by admin
  # Verify chain of trust of OCSP response using Root CA and Intermediate certs
  ssl_trusted_certificate /etc/letsencrypt/live/oddcake.net/chain.pem; # Managed by admin
  # Perfect Forward Secrecy Diffie-Hellman 4098 parameters
  ssl_dhparam /etc/ssl/private/dhparams4096.pem; # Managed by admin

  # Mozilla Modern Compatibilty SSL configuration with OCSP stapling turned on and strict settings for A+ SSL Security rating
  ssl_protocols TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLERequires nginx >= 1.13.0 else use TLSv1.2 # Dropping TLSv1.1 for modern compatability.
  ssl_ciphers 'TLS_AES_128_GCM_SHA256:TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
  ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0
  ssl_prefer_server_ciphers on;
  ssl_session_timeout 1d;

    ##
    # Nginx Bad Bot Blocker Includes
    # REPO: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker
    ##
    include /etc/nginx/bots.d/ddos.conf;
    include /etc/nginx/bots.d/blockbots.conf;

  ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
  ssl_session_tickets off;
  # OCSP stapling
  ssl_stapling on;
  ssl_stapling_verify on;
  resolver 1.1.1.1 1.0.0.1;

  ##
  # Harden nginx against DDOS

  ##
  # Pestmeester.nl
  client_header_timeout 10;
  client_body_timeout   10;
  keepalive_timeout     10;
  send_timeout          10;

  # Error & Access Logs
  access_log /var/log/nginx/oddcake.net.access.log netdata;
  error_log /var/log/nginx/oddcake.net.error.log warn;

  # Linuxbabe
  index index.php index.html index.htm index.nginx-debian.html;

  # Linuxbabe
  location / {
    try_files $uri $uri/ /index.php;
  }

  # Non-Linuxbabe
  #location / {
    #try_files $uri $uri/ /index.php?$args;
  #}

  # Non-Linuxbabe
  location ~ /.well-known {
    allow all;
  }

  # Non-Linuxbabe
  location = /favicon.ico {
    log_not_found off;
    access_log off;
  }

  # Non-Linuxbabe
  location = /robots.txt {
    allow all;
    log_not_found off;
    access_log off;
  }

  ## Configure browser caching for up to 1 year for the Hummingbird WP Plugin
  #location ~* \.(txt|xml|js)$ {
    #expires 1y;
  #}
  #location ~* \.(css)$ {
    #expires 1y;
  #}
  #location ~* \.(flv|ico|pdf|avi|mov|ppt|doc|mp3|wmv|wav|mp4|m4v|ogg|webm|aac|eot|ttf|otf|woff|woff2|svg)$ {
    #expires 1y;
  #}
  #location ~* \.(jpg|jpeg|png|gif|swf|webp)$ {
    #expires 1y;
  #}

  # Non-LinuxBabe (https://tonyteaches.tech/wepb-images-nginx/)
  # serve png, jpg, jpeg as webp if available
  #location ~* ^.+\.(png|jpe?g)$ {
    #add_header Vary Accept;
    #try_files $uri$webp_suffix $uri =404;
  #}

  # Linuxbabe
  location ~ ^/wp-json/ {
    rewrite ^/wp-json/(.*?)$ /?rest_route=/$1 last;
  }

  # Linuxbabe
  location ~* /wp-sitemap.*\.xml {
    try_files $uri $uri/ /index.php$is_args$args;
  }

  # Linuxbabe
  error_page 404 /404.html;
  error_page 500 502 503 504 /50x.html;

  # Linuxbabe
  location = /50x.html {
    root /usr/share/nginx/html;
  }

  ##
  # Pagespeed
  ##

  # enable pagespeed module on this server block
  pagespeed on;
  pagespeed Domain https://www.oddcake.net;
  pagespeed Domain https://oddcake.net;
  # Include UNIVERSAL Settings for Pagespeed Module
  include /etc/nginx/pagespeed.conf;

  # Non-Linuxbabe
  # Block access to xmlrpc.php file
  location ~ /xmlrpc.php$ {
    allow 192.168.30.0/24;
    deny all;
    #error_page 403 =444;
    include snippets/fastcgi-php.conf;
    fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
  }

  # Non-Linuxbabe
  # Only allow access of /admin via internal IP
  location ^~ /admin/ {
    allow 192.168.30.0/24;
    deny all;
    error_page 403 =444;
    include snippets/fastcgi-php.conf;
    fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
  }

  # Non-Linuxbabe
  # Allow local only to wp-login page
  location ~ /wp-login.php {
    allow 192.168.30.0/24;
    deny all;
    error_page 403 =444;
    include snippets/fastcgi-php.conf;
    fastcgi_pass unix:/var/run/php/php7.4-fpm.sock;
  }

  # Non-Linuxbabe
  # Disallow PHP In Upload Folder
  location /wp-content/uploads/ {
    location ~ \.php$ {
    deny all;
    }
  }

  # Non-Linuxbabe
  # Security Settings For Better Privacy Deny Hidden Files
  location ~ /\. {
    deny all;
    access_log off;
    log_not_found off;
  }

  # Linuxbabe - disable access to hidden files
  location ~ /\.ht {
      access_log off;
      log_not_found off;
      deny all;
  }

  ##
  # FastCGI Cache
  ##

  # Cache Purging should be restricted to allowed IP addresses.
  # If not set, an attacker may be able to wipe your nginx fastcgi cache using simple GET requests. Linuxbabe (User Comment).
  # This (supposedly) enables cache purge but restricts it to your ip address and to your loopback address.
  location ~ /purge(/.*) {
    allow 127.0.0.1;
    allow 149.28.125.6; # Replace with your servers IPv4 address
    deny all;
    # Enable http-cache-purge module in nginx for above IP addresses
    fastcgi_cache_purge phpcache "$scheme$request_method$host$1";
  }

##
# START SKIP CACHE RULES: Prevent FastCGI caching for certain things/pages
##

set $skip_cache 0;

# POST requests and urls with a query string should always go to PHP
if ($request_method = POST) {
    set $skip_cache 1;
}
if ($query_string != "") {
    set $skip_cache 1;
}

# Don't cache uris containing the following segments
if ($request_uri ~* "/wp-admin/|/xmlrpc.php|wp-.*.php|^/feed/*|/tag/.*/feed/*|index.php|/.*sitemap.*\.(xml|xsl)") {
    set $skip_cache 1;
}

# Don't use the cache for logged in users or recent commenters
if ($http_cookie ~* "comment_author|wordpress_[a-f0-9]+|wp-postpass|wordpress_no_cache|wordpress_logged_in") {
    set $skip_cache 1;
}

# Sometimes I want to test the upstream (PHP-FPM and MariaDB) response time, so I also add the following lines to tell Nginx to bypass the FastCGI cache for my own IP addresses.
# Note: This was not fullly understood from linuxbabe tutorial fastcgi chapter, and therefore was modified by admin. Use carefully.
#if ($remote_addr ~* "108.231.125.253|108.231.125.253") {
#     set $skip_cache 1;
#}

# Skip the fastCGI Cache for ATNT Public IP, and the local network.
#if ($remote_addr ~* "108.231.125.253|108.231.125.254|192.168.30..*) {
#    set $skip_cache 1;
#}

# THIS IS THE MODIFIED CODE REFERENCED in 2 sections above from here.
# Lets deactivate it for now so that we don't bypass FastCGI Cache for our own IP, or fuck things up with modified lines.

#if ($remote_addr ~* "108.231.125.253|192.168.30..*") {
#    set $skip_cache 1;
#}

# IMPORTANT NOTE WITH SEO:
# If you use the Yoast SEO or Google XML Sitemap plugins to generate sitemap, then
# you also need to move the Yoast/Google XML rewrite rules below the skip cache rules.

##
# END SKIP CACHE RULES: Prevent FastCGI caching for certain things/pages
##

  # Linuxbabe
  # Pass Fastcgi to php
  location ~ \.php$ {
    fastcgi_pass unix:/run/php/php7.4-fpm.sock;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include fastcgi_params;
    include snippets/fastcgi-php.conf;
    # Linuxbabe - FastCGI
    fastcgi_cache phpcache;
    fastcgi_cache_valid 200 301 302 60m;
    fastcgi_cache_use_stale error timeout updating invalid_header http_500 http_503;
    fastcgi_cache_min_uses 1;
    fastcgi_cache_lock on;
    # Tell Nginx to send request to upstream PHP-FPM server, instead of trying to find files in the cache.
    fastcgi_cache_bypass $skip_cache;
    # Tell Nginx not to cache the response.
    fastcgi_no_cache $skip_cache;
    add_header X-FastCGI-Cache $upstream_cache_status;
    add_header X-Content-Type-Options nosniff;
    ##
    # Harden nginx against DDOS
    ##
    # Pestmeester.nl
    #client_header_timeout 10;
    client_body_timeout   10;
    keepalive_timeout     10 10;
    send_timeout          10;
  }

  # Linuxbabe (Mostly, added some files extensions)
  # A long browser cache lifetime can speed up repeat visits to your page
  location ~* \.(jpg|ogg|ogv|svgz|eot|otf|mp4|rss|atom|zip|tgz|gz|rar|bz2|doc|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf|jpeg|gif|png|webp|svg|woff|woff2|ttf|css|js|ico|xml)$ {
    access_log        off;
    log_not_found     off;
    expires           360d;
  }
}

Server (please complete the following information):

uname -a
Linux 5.4.0-1050-raspi #56-Ubuntu SMP PREEMPT Thu Jan 13 13:09:35 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

nginx -v
nginx version: nginx/1.21.6
mitchellkrogza commented 2 years ago

Hi @Danrancan you can either comment out the duplicate settings in /etc/nginx/conf.d/botblocker-nginx-settings.conf or in your nginx.conf it's up to you.

Danrancan commented 2 years ago

Hi @Danrancan you can either comment out the duplicate settings in /etc/nginx/conf.d/botblocker-nginx-settings.conf or in your nginx.conf it's up to you.

Thanks for the reply. I am aware that I can just comment this out. But that doesn't really answer the question or solve the issue, because as soon as I comment it out, I then get another nginx -t error about another duplicate line that is non-exhistent, and after I comment that out, another duplicate error and so forth in a cycle of duplicate errors when the duplicates don't seem to exist in the first place. What I am trying to say, is that the grep search I did is somehow showing that my nginx modules are creating duplicate directives that clash with botblocker. There is no other trace of these duplicate lines anywhere in my conf files. So my question is, HOW CAN I FIND THE REAL DUPLICATE or, TRACE WHAT IS CAUSING NGINX TO THINK I HAVE A DUPLICATE, WHEN I REALLY DO NOT?

mitchellkrogza commented 2 years ago

If you commented out all those variables in botblocker-nginx-settings.conf as below and it still gives you an error

# Version 1.1
# ! new directives also to be added to include_filelist.txt ! #

#server_names_hash_bucket_size 256;
#server_names_hash_max_size 4096;
#variables_hash_max_size 4096;
#variables_hash_bucket_size 4096;
limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;
limit_conn_zone $binary_remote_addr zone=addr:50m;

then it cannot be the blocker but rather these files where your grep hit matches 

Binary file /etc/nginx/modules/ngx_http_brotli_static_module.so matches
Binary file /etc/nginx/modules/ngx_http_modsecurity_module.so matches
Binary file /etc/nginx/modules/ngx_pagespeed.so matches
Danrancan commented 2 years ago

then it cannot be the blocker but rather these files where your grep hit matches

@mitchellkrogza I suspected that these files were somehow causing the issue since grep hits them. However, I have no idea what that means. Does it somehow mean that Binary file /etc/nginx/modules/ngx_http_brotli_static_module.so is somehow magically inserting invisible and duplicate entries in my conf files? Do these .so modules somehow have code or a file inside of them that enters duplicate entries in my conf files when using the modules? I'm literally so confused here. I just don't understand where these duplicates are coming from when there is no visible or apparent duplicate in any of my conf files. P.S. Sorry for the late reply.

fizzyduck commented 2 years ago

@Danrancan Just a thought but I can see these lines in your config:

    include /etc/nginx/conf.d/botblocker-nginx-settings.conf;
    include /etc/nginx/conf.d/globalblacklist.conf;
[lines removed]
    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;

so it might be inadvertendly including everything twice. You could probably remove the first 2 lines as the 2nd set should also include the first. It's probably worth checking this.