search

mitchellkrogza / nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail for Repeat Offenders
Other
3.92k stars 475 forks source link

[CLOUDFLARE] Wrong Curl -I results, differ from test. #276

Closed mrEckendonk closed 5 years ago

mrEckendonk commented 5 years ago

Describe the bug

curl -I Gives strange information when testing twice. First time it gives curl: (56) TCP connection reset by peer Second time it gives HTTP/1.1 520 Origin Error Test made on other server with 2 different domains

A clear and concise description of what the bug is.

To Reproduce

Steps to reproduce the behavior:

[13:56][root@server7.[private] html]# curl -I https://[domain1] -e http://100dollars-seo.com
curl: (56) TCP connection reset by peer
[13:56][root@server7.[private] html]# curl -I https://[domain2] -e http://100dollars-seo.com        
HTTP/1.1 520 Origin Error
Date: Fri, 28 Jun 2019 13:56:39 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Set-Cookie: __cfduid=d3c6405293d325808c3130fd81d90d3761561730199; expires=Sat, 27-Jun-20 13:56:39 GMT; path=/; domain=.[domain2]; HttpOnly; Secure
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Set-Cookie: cf_use_ob=0; path=/; expires=Fri, 28-Jun-19 13:57:09 GMT
Expires: Thu, 01 Jan 1970 00:00:01 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Server: cloudflare
CF-RAY: 4ee02a558935bdc3-AMS

[13:56][root@server7.[private] html]# curl -I https://[domain2] -e http://100dollars-seo.com
HTTP/1.1 520 Origin Error
Date: Fri, 28 Jun 2019 13:58:50 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Set-Cookie: __cfduid=d5562662f6b09696b3391199d72472a111561730330; expires=Sat, 27-Jun-20 13:58:50 GMT; path=/; domain=.[domain2]; HttpOnly; Secure
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Set-Cookie: cf_use_ob=0; path=/; expires=Fri, 28-Jun-19 13:59:20 GMT
Expires: Thu, 01 Jan 1970 00:00:01 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Server: cloudflare
CF-RAY: 4ee02d839ec69d60-AMS

[13:58][root@server7.[private] html]# curl -I https://[domain1] -e http://100dollars-seo.com
HTTP/1.1 520 Origin Error
Date: Fri, 28 Jun 2019 13:58:56 GMT
Content-Type: text/html; charset=UTF-8
Connection: keep-alive
Set-Cookie: __cfduid=d99ee2f4f38f0e9fe504b5874b5e253881561730336; expires=Sat, 27-Jun-20 13:58:56 GMT; path=/; domain=.[domain1]; HttpOnly
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Set-Cookie: cf_use_ob=0; path=/; expires=Fri, 28-Jun-19 13:59:26 GMT
Expires: Thu, 01 Jan 1970 00:00:01 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Server: cloudflare
CF-RAY: 4ee02dab0b19d8b9-AMS

Expected behavior

Getting correct curl: (56) TCP connection reset by peer instead of 202

Copy of nginx.conf

If applicable please paste your nginx.conf file here (paste in between the markers)

user              nginx nginx;

worker_processes 2;
worker_priority -10;

worker_rlimit_nofile 520000;
timer_resolution 100ms;

pcre_jit on;
include /usr/local/nginx/conf/dynamic-modules.conf;

pid         logs/nginx.pid;

events {
    worker_connections  80000;
    accept_mutex off;
    accept_mutex_delay 200ms;
    use epoll;
    #multi_accept on;
}

http { 
limit_req_zone $binary_remote_addr zone=xwprpc:10m rate=30r/s;

 include /usr/local/nginx/conf/brotli_inc.conf;
 map_hash_bucket_size 128;
 map_hash_max_size 4096;
 server_names_hash_bucket_size 128;
 server_names_hash_max_size 4096;
 variables_hash_max_size 2048;

# Bad Bot Blocker
  include /usr/local/nginx/conf/ultimate-badbot-blocker/botblocker-nginx-settings.conf;
  include /usr/local/nginx/conf/ultimate-badbot-blocker/globalblacklist.conf;

 resolver 127.0.0.1 valid=10m;
 resolver_timeout 10s;

limit_req_zone $binary_remote_addr zone=xwplogin:16m rate=40r/m;
#limit_conn_zone $binary_remote_addr zone=xwpconlimit:16m;

# sets Centmin Mod headers via headers more nginx module
# https://github.com/openresty/headers-more-nginx-module
# don't remove the first 2 lines as centmin mod checks to see if they're 
# missing and re-adds them anyway. Just uncomment the 3rd & 4th lines
# which is used to override the Server header to what you want = nginx
# and remove the X-Powered-By header + restart nginx service
# do not disable headers more nginx module itself as it's required for
# other centmin mod features like redis nginx level caching & letsencrypt
# integration in vhosts created by addons/acmetool.sh
more_set_headers "Server: nginx centminmod";
more_set_headers "X-Powered-By: centminmod";
#more_set_headers "Server: nginx";
#more_clear_headers "X-Powered-By";

# uncomment cloudflare.conf include if using cloudflare for
# server and/or vhost site + setup cron job for command
# /usr/local/src/centminmod/tools/csfcf.sh auto
# run the auto command once to populate cloudflare ips
#include /usr/local/nginx/conf/cloudflare.conf;
# uncomment incapsula.conf include if using incapsula for
# server and/or vhost site + setup cron job for command
# /usr/local/src/centminmod/tools/csfincapsula.sh auto
# run the auto command once to popular incapsula ips
#include /usr/local/nginx/conf/incapsula.conf;
include /usr/local/nginx/conf/maintenance.conf;
#include /usr/local/nginx/conf/vts_http.conf;
include /usr/local/nginx/conf/geoip.conf;
include /usr/local/nginx/conf/webp.conf;
include /usr/local/nginx/conf/ssl_include.conf;
#include /usr/local/nginx/conf/pagespeedadmin.conf;
include /usr/local/nginx/conf/fastcgi_param_https_map.conf;
include /usr/local/nginx/conf/redisupstream.conf;
include /usr/local/nginx/conf/wpcacheenabler_map.conf;
include /usr/local/nginx/conf/wpfastcgi_cache_map.conf;
include /usr/local/nginx/conf/wpfastcgi_cache_path_includes.conf;
include /usr/local/nginx/conf/default_phpupstream.conf;

log_format  main  '$remote_addr - $remote_user [$time_local] $request '
                '"$status" $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                ' "$connection" "$connection_requests" "$request_time"';

log_format  ddos-proxy '$remote_addr for $http_x_real_ip - $remote_user [$time_local] $request '
                '"$status" $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio"'
                ' "$connection" "$connection_requests" "$request_time"';

log_format  main_ext  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" '
                      '"$host" sn="$server_name" '
                      'rt=$request_time '
                      'ua="$upstream_addr" us="$upstream_status" '
                      'ut="$upstream_response_time" ul="$upstream_response_length" '
                      'cs=$upstream_cache_status' ;

# only uncomment include line to enable it you have enabled ngx_brotli module
# which is disabled by default https://community.centminmod.com/threads/10688/
#include /usr/local/nginx/conf/log_format_brotli.conf;

access_log  off;
error_log   logs/error.log warn;

    index  index.php index.html index.htm;
    include       mime.types;
    default_type  application/octet-stream;
    charset utf-8;

sendfile on;
sendfile_max_chunk 512k;
tcp_nopush  on;
tcp_nodelay on;
server_tokens off;
server_name_in_redirect off;

keepalive_timeout  5s;
keepalive_requests 1000;
lingering_time 20s;
lingering_timeout 5s;
keepalive_disable msie6;

gzip on;
gzip_vary   on;
gzip_disable msie6;
gzip_static on;
gzip_min_length   1400;
gzip_buffers      1024 8k;
gzip_http_version 1.1;
gzip_comp_level 5;
gzip_proxied    any;
gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml application/xml+rss application/ecmascript application/json image/svg+xml;

client_body_buffer_size 256k;
client_body_in_file_only off;
client_body_timeout 10s;
client_header_buffer_size 64k;
## how long a connection has to complete sending 
## it's headers for request to be processed
client_header_timeout  10s;
client_max_body_size 1024m; 
connection_pool_size  512;
directio  4m;
directio_alignment 4096;
ignore_invalid_headers on;       
large_client_header_buffers 8 64k;
output_buffers   1 512k;
postpone_output  1460;
proxy_temp_path  /tmp/nginx_proxy/;
request_pool_size  32k;
reset_timedout_connection on;
send_timeout     60s;
types_hash_max_size 2048;

# for nginx proxy backends to prevent redirects to backend port 
# port_in_redirect off;

open_file_cache max=50000 inactive=60s;
open_file_cache_valid 120s;
open_file_cache_min_uses 2;
open_file_cache_errors off;
open_log_file_cache max=10000 inactive=30s min_uses=2;

 include /usr/local/nginx/conf/conf.d/*.conf;
}

domain1

Copy of vhost / website / host .conf file


#x# HTTPS-DEFAULT
 server {

   server_name [domain1] www.[domain1];
   return 302 https://[domain1]$request_uri;
   include /usr/local/nginx/conf/staticfiles.conf;
 }

server {
  listen 443 ssl http2;
  server_name [domain1] www.[domain1];

  include /usr/local/nginx/conf/ssl/[domain1]/[domain1].crt.key.conf;
  include /usr/local/nginx/conf/ssl_include.conf;

  # cloudflare authenticated origin pull cert community.centminmod.com/threads/13847/
  #ssl_client_certificate /usr/local/nginx/conf/ssl/cloudflare/[domain1]/origin.crt;
  #ssl_verify_client on;
  http2_max_field_size 16k;
  http2_max_header_size 32k;
  # mozilla recommended
  ssl_ciphers TLS13-AES-128-GCM-SHA256:TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-128-CCM-8-SHA256:TLS13-AES-128-CCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
  ssl_prefer_server_ciphers   on;
  #add_header Alternate-Protocol  443:npn-spdy/3;

  # before enabling HSTS line below read centminmod.com/nginx_domain_dns_setup.html#hsts
  #add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
  #add_header X-Frame-Options SAMEORIGIN;
  add_header X-Xss-Protection "1; mode=block" always;
  add_header X-Content-Type-Options "nosniff" always;
  #add_header Referrer-Policy "strict-origin-when-cross-origin";
  #spdy_headers_comp 5;
  ssl_buffer_size 1369;
  ssl_session_tickets on;

  # enable ocsp stapling
  resolver 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 valid=10m;
  resolver_timeout 10s;
  ssl_stapling on;
  ssl_stapling_verify on;

# ngx_pagespeed & ngx_pagespeed handler
#include /usr/local/nginx/conf/pagespeed.conf;
#include /usr/local/nginx/conf/pagespeedhandler.conf;
#include /usr/local/nginx/conf/pagespeedstatslog.conf;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/[domain1]/log/access.log combined buffer=256k flush=5m;
  error_log /home/nginx/domains/[domain1]/log/error.log;

  include /usr/local/nginx/conf/autoprotect/[domain1]/autoprotect-[domain1].conf;
  root /home/nginx/domains/[domain1]/public;
  # uncomment cloudflare.conf include if using cloudflare for
  # server and/or vhost site
  include /usr/local/nginx/conf/cloudflare.conf;
  include /usr/local/nginx/conf/503include-main.conf;

  #include /usr/local/nginx/conf/wpincludes/[domain1]/wpfastcgicache_include_[domain1].conf;
  include /usr/local/nginx/conf/wpincludes/[domain1]/wpcacheenabler_[domain1].conf;
  #include /usr/local/nginx/conf/wpincludes/[domain1]/wpsupercache_[domain1].conf;
  # https://community.centminmod.com/posts/18828/
  #include /usr/local/nginx/conf/wpincludes/[domain1]/rediscache_[domain1].conf;  

  location / {
  include /usr/local/nginx/conf/503include-only.conf;

  # Bad Bot Blocker
  include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;
  include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;

  # Enables directory listings when index file not found
  #autoindex  on;

  # for wordpress super cache plugin
  #try_files /wp-content/cache/supercache/$http_host/$cache_uri/index.html $uri $uri/ /index.php?q=$uri&$args;

  # for wp cache enabler plugin
  try_files $cache_enabler_uri $uri $uri/ $custom_subdir/index.php?$args;  

  # Wordpress fastcgi_cache
    #try_files $uri $uri/ /index.php?$args;

  # Wordpress Permalinks
   #try_files $uri $uri/ /index.php?q=$uri&$args;

  # Nginx level redis Wordpress
  # https://community.centminmod.com/posts/18828/
  #try_files $uri $uri/ /index.php?$args;

  }

location ~* /(wp-login\.php) {
    limit_req zone=xwplogin burst=1 nodelay;
    #limit_conn xwpconlimit 30;
    auth_basic "Private";
    auth_basic_user_file /home/nginx/domains/[domain1]/htpasswd_wplogin;    
    include /usr/local/nginx/conf/php-wpsc.conf;

    # https://community.centminmod.com/posts/18828/
    #include /usr/local/nginx/conf/php-rediscache.conf;
}

location ~* /(xmlrpc\.php) {
    limit_req zone=xwprpc burst=45 nodelay;
    #limit_conn xwpconlimit 30;
    include /usr/local/nginx/conf/php-wpsc.conf;

    # https://community.centminmod.com/posts/18828/
    #include /usr/local/nginx/conf/php-rediscache.conf;
}

location ~* /wp-admin/(load-scripts\.php) {
    limit_req zone=xwprpc burst=5 nodelay;
    #limit_conn xwpconlimit 30;
    include /usr/local/nginx/conf/php-wpsc.conf;

    # https://community.centminmod.com/posts/18828/
    #include /usr/local/nginx/conf/php-rediscache.conf;
}

location ~* /wp-admin/(load-styles\.php) {
    limit_req zone=xwprpc burst=5 nodelay;
    #limit_conn xwpconlimit 30;
    include /usr/local/nginx/conf/php-wpsc.conf;

    # https://community.centminmod.com/posts/18828/
    #include /usr/local/nginx/conf/php-rediscache.conf;
}

  include /usr/local/nginx/conf/wpincludes/[domain1]/wpsecure_[domain1].conf;
  include /usr/local/nginx/conf/php-wpsc.conf;

  # https://community.centminmod.com/posts/18828/
  #include /usr/local/nginx/conf/php-rediscache.conf;
  include /usr/local/nginx/conf/pre-staticfiles-local-[domain1].conf;
  include /usr/local/nginx/conf/pre-staticfiles-global.conf;
  include /usr/local/nginx/conf/staticfiles.conf;
  include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  include /usr/local/nginx/conf/vts_server.conf;
}

domain2

Copy of vhost / website / host .conf file

## Server (please complete the following information):
 #x# HTTPS-DEFAULT
 server {

   server_name [domain2] www.[domain2];
   return 302 https://[domain2]$request_uri;
   include /usr/local/nginx/conf/staticfiles.conf;
 }

server {
  listen 443 ssl http2;
  server_name [domain2] www.[domain2];

  include /usr/local/nginx/conf/ssl/[domain2]/[domain2].crt.key.conf;
  include /usr/local/nginx/conf/ssl_include.conf;

  # cloudflare authenticated origin pull cert community.centminmod.com/threads/13847/
  #ssl_client_certificate /usr/local/nginx/conf/ssl/cloudflare/[domain2]/origin.crt;
  #ssl_verify_client on;
  http2_max_field_size 16k;
  http2_max_header_size 32k;
  # mozilla recommended
  ssl_ciphers TLS13-AES-128-GCM-SHA256:TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-128-CCM-8-SHA256:TLS13-AES-128-CCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;
  ssl_prefer_server_ciphers   on;
  #add_header Alternate-Protocol  443:npn-spdy/3;

  # before enabling HSTS line below read centminmod.com/nginx_domain_dns_setup.html#hsts
  #add_header Strict-Transport-Security "max-age=31536000; includeSubdomains;";
  #add_header X-Frame-Options SAMEORIGIN;
  add_header X-Xss-Protection "1; mode=block" always;
  add_header X-Content-Type-Options "nosniff" always;
  #add_header Referrer-Policy "strict-origin-when-cross-origin";
  #spdy_headers_comp 5;
  ssl_buffer_size 1369;
  ssl_session_tickets on;

  # enable ocsp stapling
  resolver 8.8.8.8 8.8.4.4 1.1.1.1 1.0.0.1 valid=10m;
  resolver_timeout 10s;
  ssl_stapling on;
  ssl_stapling_verify on;

# ngx_pagespeed & ngx_pagespeed handler
#include /usr/local/nginx/conf/pagespeed.conf;
#include /usr/local/nginx/conf/pagespeedhandler.conf;
#include /usr/local/nginx/conf/pagespeedstatslog.conf;

  # limit_conn limit_per_ip 16;
  # ssi  on;

  access_log /home/nginx/domains/[domain2]/log/access.log combined buffer=256k flush=5m;
  error_log /home/nginx/domains/[domain2]/log/error.log;

  include /usr/local/nginx/conf/autoprotect/[domain2]/autoprotect-[domain2].conf;
  root /home/nginx/domains/[domain2]/public;
  # uncomment cloudflare.conf include if using cloudflare for
  # server and/or vhost site
  include /usr/local/nginx/conf/cloudflare.conf;
  include /usr/local/nginx/conf/503include-main.conf;

  #include /usr/local/nginx/conf/wpincludes/[domain2]/wpfastcgicache_include_[domain2].conf;
  include /usr/local/nginx/conf/wpincludes/[domain2]/wpcacheenabler_[domain2].conf;
  #include /usr/local/nginx/conf/wpincludes/[domain2]/wpsupercache_[domain2].conf;
  # https://community.centminmod.com/posts/18828/
  #include /usr/local/nginx/conf/wpincludes/[domain2]/rediscache_[domain2].conf;  

  location / {
  include /usr/local/nginx/conf/503include-only.conf;

  # Bad Bot Blocker
  include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/ddos.conf;
  include /usr/local/nginx/conf/ultimate-badbot-blocker/bots.d/blockbots.conf;

  # Enables directory listings when index file not found
  #autoindex  on;

  # for wordpress super cache plugin
  #try_files /wp-content/cache/supercache/$http_host/$cache_uri/index.html $uri $uri/ /index.php?q=$uri&$args;

  # for wp cache enabler plugin
  try_files $cache_enabler_uri $uri $uri/ $custom_subdir/index.php?$args;  

  # Wordpress fastcgi_cache
    #try_files $uri $uri/ /index.php?$args;

  # Wordpress Permalinks
   #try_files $uri $uri/ /index.php?q=$uri&$args;

  # Nginx level redis Wordpress
  # https://community.centminmod.com/posts/18828/
  #try_files $uri $uri/ /index.php?$args;

  }

location ~* /(wp-login\.php) {
    limit_req zone=xwplogin burst=1 nodelay;
    #limit_conn xwpconlimit 30;
    auth_basic "Private";
    auth_basic_user_file /home/nginx/domains/[domain2]/htpasswd_wplogin;    
    include /usr/local/nginx/conf/php-wpsc.conf;

    # https://community.centminmod.com/posts/18828/
    #include /usr/local/nginx/conf/php-rediscache.conf;
}

location ~* /(xmlrpc\.php) {
    limit_req zone=xwprpc burst=45 nodelay;
    #limit_conn xwpconlimit 30;
    include /usr/local/nginx/conf/php-wpsc.conf;

    # https://community.centminmod.com/posts/18828/
    #include /usr/local/nginx/conf/php-rediscache.conf;
}

location ~* /wp-admin/(load-scripts\.php) {
    limit_req zone=xwprpc burst=5 nodelay;
    #limit_conn xwpconlimit 30;
    include /usr/local/nginx/conf/php-wpsc.conf;

    # https://community.centminmod.com/posts/18828/
    #include /usr/local/nginx/conf/php-rediscache.conf;
}

location ~* /wp-admin/(load-styles\.php) {
    limit_req zone=xwprpc burst=5 nodelay;
    #limit_conn xwpconlimit 30;
    include /usr/local/nginx/conf/php-wpsc.conf;

    # https://community.centminmod.com/posts/18828/
    #include /usr/local/nginx/conf/php-rediscache.conf;
}

  include /usr/local/nginx/conf/wpincludes/[domain2]/wpsecure_[domain2].conf;
  include /usr/local/nginx/conf/php-wpsc.conf;

  # https://community.centminmod.com/posts/18828/
  #include /usr/local/nginx/conf/php-rediscache.conf;
  include /usr/local/nginx/conf/pre-staticfiles-local-[domain2].conf;
  include /usr/local/nginx/conf/pre-staticfiles-global.conf;
  include /usr/local/nginx/conf/staticfiles.conf;
  include /usr/local/nginx/conf/drop.conf;
  #include /usr/local/nginx/conf/errorpage.conf;
  include /usr/local/nginx/conf/vts_server.conf;
}

Linux server8.[private] 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux


 - Nginx Version [post output of sudo nginx -v

[16:45][root@server8.[private] conf.d]# nginx -V nginx version: nginx/1.17.1 (270619-130104-centos7-kvm) built by gcc 8.2.1 20180905 (Red Hat 8.2.1-3) (GCC) built with OpenSSL 1.1.1c 28 May 2019 TLS SNI support enabled configure arguments: --with-ld-opt='-Wl,-E -L/usr/local/zlib-cf/lib -L/usr/local/lib -ljemalloc -lpcre -Wl,-z,relro -Wl,-rpath,/usr/local/zlib-cf/lib:/usr/local/lib' --with-cc-opt='-I/usr/local/zlib-cf/include -I/usr/local/include -m64 -march=native -DTCP_FASTOPEN=23 -g -O3 -Wno-error=strict-aliasing -fstack-protector-strong -flto --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wimplicit-fallthrough=0 -fcode-hoisting -Wp,-D_FORTIFY_SOURCE=2 -Wno-deprecated-declarations -gsplit-dwarf' --sbin-path=/usr/local/sbin/nginx --conf-path=/usr/local/nginx/conf/nginx.conf --build=270619-130104-centos7-kvm --with-compat --with-http_stub_status_module --with-http_secure_link_module --with-libatomic --with-http_gzip_static_module --add-dynamic-module=../ngx_brotli --add-dynamic-module=../ngx_http_geoip2_module --with-http_sub_module --with-http_addition_module --with-http_image_filter_module=dynamic --with-http_geoip_module --with-stream_geoip_module --with-stream_realip_module --with-stream_ssl_preread_module --with-threads --with-stream --with-stream_ssl_module --with-http_realip_module --add-dynamic-module=../ngx-fancyindex-0.4.2 --add-module=../ngx_cache_purge-2.5 --add-dynamic-module=../ngx_devel_kit-0.3.0 --add-dynamic-module=../set-misc-nginx-module-0.32 --add-dynamic-module=../echo-nginx-module-0.61 --add-module=../redis2-nginx-module-0.15 --add-module=../ngx_http_redis-0.3.7 --add-dynamic-module=../lua-nginx-module-0.10.15 --add-module=../stream-lua-nginx-module-0.0.7 --add-module=../memc-nginx-module-0.18 --add-module=../srcache-nginx-module-0.31 --add-dynamic-module=../headers-more-nginx-module-0.33 --with-pcre-jit --with-zlib=../zlib-cloudflare-1.3.0 --with-http_ssl_module --with-http_v2_module --with-openssl=../openssl-1.1.1c --with-openssl-opt='enable-ec_nistp_64_gcc_128 enable-tls1_3' --add-dynamic-module=../ModSecurity-nginx



 - Other Environments [include Version information]

 - [ X] Other
 - Specify Other / Specific Version Information Here: 

Centmin Mod Menu 123.09beta01 centminmod.com
mitchellkrogza commented 5 years ago

Can you try testing with another referrer like zx6.ru to see if Cloudflare has some kind of prevention for what 100dollars-seo might be doing?

mitchellkrogza commented 5 years ago

Unfortunately I do not use Cloudflare. But here you can see your problem is entirely Cloudflare related.

$ curl -I https://[domain1] -e http://100dollars-seo.com
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
$ curl -I https://[domain2] -e http://100dollars-seo.com
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
$ curl -I https://[domain3] -e http://100dollars-seo.com
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
$ curl -I https://[domain4] -e http://100dollars-seo.com
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
$ curl -I https://[domain1] -e http://100dollars-seo.com
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
mrEckendonk commented 5 years ago

Thanks for your info @mitchellkrogza disabling Cloudflare caching resolved it. Using it as DNS and some basic functions is fine. FYI tested with Nginx 1.17,1 so you can update you README.md

So keep with one question. curl -A with the first 2 test gives pure html, not code 200 OK. Does this mean it works. Thats a little unclear in the README.md

The other gives the correct as indicated

[19:55][root@server7.{private] html]# curl -I https://[domain1] -e http://100dollars-seo.com
curl: (56) TCP connection reset by peer
[19:58][root@server7.[private] html]# curl -I https://[domain1] -e http://zx6.ru                     
curl: (56) TCP connection reset by peer
[20:02][root@server7.[private] html]# curl -A "Xenu Link Sleuth/1.3.8"  https://[domain1]
curl: (56) TCP connection reset by peer
[20:03][root@server7.[private} html]# curl -A "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)" https://[domain1]
curl: (56) TCP connection reset by peer
mitchellkrogza commented 5 years ago

I am busy with so many changes including updating documentation a few hundred commits this week :scream: It should just be a curl -A "whatever" -I https://your domain.com for quicker testing without the full code response

mrEckendonk commented 5 years ago

Thanks, my email box is full with you commits. Keep up the good work @mitchellkrogza 

curl -A "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" -I https://[domain1] 
HTTP/1.1 200 OK
mrEckendonk commented 5 years ago

Sorry, did not see your request "Can you try testing with another referrer like zx6.ru", but that does not make any difference. Until now, we need to disable Cloudflare caching.

mitchellkrogza commented 5 years ago

No ignore that it's purely just a Cloudflare thing, I just wasnt sure if they had some kind of other protection to detect naughty domains which is not the case

mrEckendonk commented 5 years ago

Got a message from the maintainer of the LEMP Stack that we use.

that is expected as Cloudflare is the one to receive your 443/444 blocked status message from Centmin Mod Nginx server which CF can't read so gives the 520 error which is what you'd want for blocked requests. So no need to disable Cloudflare cache for live use as you want bad bots to receive that 520 error from CF.

I guess that this is the same for other CDN's.

mitchellkrogza commented 5 years ago

Thanks @Eckybrazzz this is very good info to have here for other users of CF. It shows CF will spare your server from being hit by a bunch of repetitive queries from the same BOT when on its first attempt you said 444 / Go Away :+1:

mrEckendonk commented 5 years ago

@mitchellkrogza, maybe an Idea to adjust your README step 10 in the meanwhile with "If using CF.... please disable the caching before testing... or something like that. I only disabled caching, and the issue was gone, I think that ALL CDN's act the same, but did not test that.

I recreated a new server because of this, thinking I did something wrong.

And a step with "suggestion" to keep the git up to date as you explain in an e-mail to me would be useful to.

#!/bin/bash
cd /path/to/nginx-ultimate-bad-bot-blocker/
git pull
# Send yourself an email - Not Necessary Though

exit ${?}
mitchellkrogza commented 5 years ago

@Eckybrazzz readme now Auto updated on each build with all different versions used during tests