search

mitchellkrogza / nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail for Repeat Offenders
Other
4.05k stars 482 forks source link

How to implement in nginx container? #194

Closed robert1112 closed 6 years ago

robert1112 commented 6 years ago

Hi

Thanks so much for awesome work. I got problem in nginx container. I ran all the commands and generate all config files but they will all disappear after reload nginx.

Should I create permanent folder and files for these config files manually (they won't go away after reload) and set up cron job to update them? Not very ideal.

Any advice for me? Thank you again.

robert1112 commented 6 years ago

Will [ -w ] : WWW path (default: /var/www) cause the failure since www is another container in the same host?

mitchellkrogza commented 6 years ago

Hi @robert1112 by container you mean? Docker? or please elaborate

itoffshore commented 6 years ago

@robert1112 - you need to use the Docker volume api for persistent storage. See also here.

lxc is a good choice for containers too & would not have these problems. An Alpine Linux lxc container with nginx is about 5-6 meg in size.

mitchellkrogza commented 6 years ago

Wow @itoffshore 5-6 Mb that's insanely tiny and impressive

robert1112 commented 6 years ago

Hi @mitchellkrogza @itoffshore Thanks so much for your advices and support on this. The Wordpress site is builded with containers on a PaaS, D2C. Here is the official stack file and here is the service structure. PHP is Wordpress and Nginx in the very front is where I run all the commands. In Nginx container, there is no such /var/www. On the platform, I can set persistent volume easily. @itoffshore Do you suggest I create them manually? Did you do that before? Will updating them be a issue? image I run install and setup commands and reload Nginx. Everything seems fine. But It didn't pass the testing described on the doc. So i think it might be the container structure or something I didnt do right. Please kindly help!! P.S. When I reload Nginx, the new installed bad bot files are not deleted.

robert1112 commented 6 years ago

@itoffshore I think this is what you refer to for Docker volume. Thank you.

image
robert1112 commented 6 years ago

Hi @mitchellkrogza @itoffshore

I ran sudo nginx -t and output successfully. But I ran the testing commands described on doc and still return 200. Do I need to install bad bot on backend nginx or reserver nginx? Here is the output of all config files. THANK YOU SO MUCH.

2018/07/29 10:10:58 [emerg] 4107#4107: unknown directive "~*\badultgalls\b" in /etc/nginx/bots.d/bad-referrer-words.conf:52
nginx: [emerg] unknown directive "~*\badultgalls\b" in /etc/nginx/bots.d/bad-referrer-words.conf:52
nginx: configuration file /etc/nginx/nginx.conf test failed

root@web2:/usr/local/sbin# sudo nginx -V

nginx version: nginx/1.15.2
built by gcc 6.3.0 20170516 (Debian 6.3.0-18+deb9u1)
built with OpenSSL 1.1.0f  25 May 2017
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.15.2/debian/debuild-base/nginx-1.15.2=. -specs=/usr/share/dpkg/no-pie-compile.specs -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-specs=/usr/share/dpkg/no-pie-link.specs -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

root@web2:/etc/nginx# cat nginx.conf

user  nginx;
worker_processes auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
  worker_connections  1024;
}

http {
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;
  server_names_hash_bucket_size 64;

  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';

  access_log  /var/log/nginx/access.log  main;

  sendfile        on;
  #tcp_nopush     on;

  keepalive_timeout  65;

  client_body_buffer_size 32K;
  client_max_body_size 10M;

  client_body_timeout 10s;
  client_header_buffer_size 1k;
  large_client_header_buffers 2 16k;
  client_header_timeout 5s;

  gzip on;
  gzip_static on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied expired no-cache no-store private auth;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json text/javascript application/x-javascript application/javascript text/xml application/xml application/xml+rss application/vnd.groove-tool-template application/octet-stream;

  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/bots.d/*.conf;
}

(Equivalent for /etc/sites-available/yoursite.conf) root@web2:/etc/nginx/conf.d# cat 5b49c3735073c91a13cf7d65.conf

server {
  listen 80;

  root /var/www/pma2;
  server_name pma2-www.2e1a7fa72c.at.d2c.io;
  index index.php index.html;
  resolver 172.17.0.1 valid=60s ipv6=off;
  set $backend pma2;
  include conf.d/letsencrypt.inc;
  location ~ /\.git {
    # Bad Bot Blocker
    include /etc/nginx/bots.d/ddos.conf;
    include /etc/nginx/bots.d/blockbots.conf;

    deny all;
  }

  try_files $uri $uri/ @rewrite;
  location @rewrite {
    rewrite ^/(.*)$ /index.php?$args;
  }
  location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|pdf|ppt|txt|bmp|rtf|js|json|tpl|ttf|woff|eot|svg)$ {
    access_log off;
    add_header Pragma public;
    add_header Cache-Control "public, must-revalidate, proxy-revalidate";
    expires 30d;
  }
  location ~ \.php$ {
    try_files $fastcgi_script_name =404;
    fastcgi_keep_conn on;
    fastcgi_pass $backend:9000;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
  }
  location ~ /\.ht {
    deny all;
  }
}

root@web2:/etc/nginx/conf.d# ls -la /etc/nginx

total 56
drwxr-xr-x 1 root root 4096 Jul 29 03:48 .
drwxr-xr-x 1 root root 4096 Jul 29 09:59 ..
drwxrwxr-x 2 root root 4096 Jul 29 10:00 bots.d
drwxrwxr-x 3 root root 4096 Jul 29 10:03 conf.d
-rw-r--r-- 1 root root 1007 Jul 24 13:02 fastcgi_params
-rw-r--r-- 1 root root 2837 Jul 24 13:02 koi-utf
-rw-r--r-- 1 root root 2223 Jul 24 13:02 koi-win
-rw-r--r-- 1 root root 5231 Jul 24 13:02 mime.types
lrwxrwxrwx 1 root root   22 Jul 24 13:02 modules -> /usr/lib/nginx/modules
-rw-r--r-- 1 root root 1279 Jul 29 03:47 nginx.conf
-rw-r--r-- 1 root root  636 Jul 24 13:02 scgi_params
-rw-r--r-- 1 root root  664 Jul 24 13:02 uwsgi_params
-rw-r--r-- 1 root root 3610 Jul 24 13:02 win-utf

root@web2:/etc/nginx/conf.d# ls -la /etc/nginx/bots.d

total 48
drwxrwxr-x 2 root root 4096 Jul 29 10:00 .
drwxr-xr-x 1 root root 4096 Jul 29 03:48 ..
-rw------- 1 root root 3566 Jul 29 10:00 bad-referrer-words.conf
-rw------- 1 root root 2450 Jul 29 10:00 blacklist-domains.conf
-rw------- 1 root root 5046 Jul 29 10:00 blacklist-ips.conf
-rw------- 1 root root 3318 Jul 29 10:00 blacklist-user-agents.conf
-rw------- 1 root root 2105 Jul 29 10:00 blockbots.conf
-rw------- 1 root root 2638 Jul 29 10:00 custom-bad-referrers.conf
-rw------- 1 root root 1768 Jul 29 10:00 ddos.conf
-rw------- 1 root root 2485 Jul 29 10:00 whitelist-domains.conf
-rw------- 1 root root 1731 Jul 29 10:01 whitelist-ips.conf

root@web2:/etc/nginx/conf.d# ls -ls /etc/nginx/conf.d

total 308
  4 -rw-r--r-- 1 root root   2235 Jul 29 10:00 5b49c36d5073c9370fcf7d60.conf
  4 -rw-r--r-- 1 root root    967 Jul 29 10:00 5b49c3735073c91a13cf7d65.conf
  4 -rw------- 1 root root   1832 Jul 29 10:03 botblocker-nginx-settings.conf
  4 -rw-r--r-- 1 root root    910 Jul 29 03:47 cloudflare.conf
284 -rw------- 1 root root 287708 Jul 29 10:00 globalblacklist.conf
  4 -rw-r--r-- 1 root root    169 Jul 29 03:47 letsencrypt.inc
  4 drwxr-xr-x 2 root root   4096 Jul 29 03:48 wordpress
robert1112 commented 6 years ago

Hi @mitchellkrogza @itoffshore I got everything done successfully but when I run test like curl -I https://wooshop.com.tw -e http://100dollars-seo.com It still return 200. Here is the conf files.

It is very close to the success. Please Kindly advice. THANK YOU SO MUCH. root@web2:/etc/nginx# cat nginx.conf

user  nginx;
worker_processes auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
  worker_connections  1024;
}

http {
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;
  #server_names_hash_bucket_size 64;

  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';

  access_log  /var/log/nginx/access.log  main;

  sendfile        on;
  #tcp_nopush     on;

  keepalive_timeout  65;

  client_body_buffer_size 32K;
  client_max_body_size 10M;

  client_body_timeout 10s;
  client_header_buffer_size 1k;
  large_client_header_buffers 2 16k;
  client_header_timeout 5s;

  gzip on;
  gzip_static on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied expired no-cache no-store private auth;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json text/javascript application/x-javascript application/javascript text/xml application/xml application/xml+rss application/vnd.groove-tool-template application/octet-stream;

  include /etc/nginx/conf.d/*.conf;
  **include /etc/nginx/global.d/botblocker-nginx-settings.conf;
  include /etc/nginx/global.d/globalblacklist.conf;**
}

root@web2:/etc/nginx/conf.d# cat 5b49c36d5073c9370fcf7d60.conf (this is the site-available)

server {
  listen 80;

  server_name wooshop.com.tw;
  include conf.d/letsencrypt.inc;
  # Redirect all HTTP requests to HTTPS with a 301 Moved Permanently response.
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl http2;

  ssl_certificate /etc/letsencrypt/live/wooshop.com.tw/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/wooshop.com.tw/privkey.pem;

  ssl_session_timeout 1d;
  ssl_session_cache shared:SSL:50m;
  ssl_session_tickets off;

  # Diffie-Hellman parameter for DHE ciphersuites
  ssl_dhparam /d2c/ssl/dhparam.pem;

  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-128-GCM-SHA256:TLS13-AES-128-CCM-8-SHA256:TLS13-AES-128-CCM-SHA256";
  ssl_prefer_server_ciphers on;
  ssl_ecdh_curve secp521r1:secp384r1:prime256v1;

  # HSTS (15768000 seconds = 6 months)
  add_header Strict-Transport-Security max-age=15768000;

  # OCSP Stapling ---
  # fetch OCSP records from URL in ssl_certificate and cache them
  ssl_stapling on;
  ssl_stapling_verify on;

  root /var/www/varnish2;
  server_name wooshop.com.tw;
  index index.html;
  resolver 172.17.0.1 valid=60s ipv6=off;
  set $backend varnish2;
  include conf.d/letsencrypt.inc;
  location ~ /\.git {
  # Bad Bot Blocker
        include /etc/nginx/bots.d/blockbots.conf;
        include /etc/nginx/bots.d/ddos.conf;

    deny all;
  }

  try_files $uri/index.html $uri @app;
  location @app {
    proxy_pass        http://$backend:80;
    proxy_redirect    off;
    proxy_set_header  Upgrade $http_upgrade;
    proxy_set_header  Connection "upgrade";
    proxy_set_header  Host $host;
    proxy_set_header  X-Real-IP $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
  }
}

root@web2:/etc/nginx/bots.d# ls

bad-referrer-words.conf  blacklist-ips.conf      blockbots.conf     ddos.conf       whitelist-ips.conf
blacklist-domains.conf   blacklist-user-agents.conf  custom-bad-referrers.conf  whitelist-domains.conf
mitchellkrogza commented 6 years ago

@robert1112 where have you placed the globalblacklist.conf file at ???

It should be located in /etc/nginx/conf.d

mitchellkrogza commented 6 years ago

You've also placed the blocker includes inside a location which will ONLY work for that location

  location ~ /\.git {
  # Bad Bot Blocker
        include /etc/nginx/bots.d/blockbots.conf;
        include /etc/nginx/bots.d/ddos.conf;

    deny all;
  }

The blocker includes must be outside of a location so it applies to the whole site so put these at somewhere like this

  # OCSP Stapling ---
  # fetch OCSP records from URL in ssl_certificate and cache them
  ssl_stapling on;
  ssl_stapling_verify on;

  # Bad Bot Blocker
        include /etc/nginx/bots.d/blockbots.conf;
        include /etc/nginx/bots.d/ddos.conf;
mitchellkrogza commented 6 years ago

@robert1112 Please try these conf files for me

nginx.conf

user  nginx;
worker_processes auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
  worker_connections  1024;
}

http {
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;
  #server_names_hash_bucket_size 64;

  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';

  access_log  /var/log/nginx/access.log  main;

  sendfile        on;
  #tcp_nopush     on;

  keepalive_timeout  65;

  client_body_buffer_size 32K;
  client_max_body_size 10M;

  client_body_timeout 10s;
  client_header_buffer_size 1k;
  large_client_header_buffers 2 16k;
  client_header_timeout 5s;

  gzip on;
  gzip_static on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied expired no-cache no-store private auth;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json text/javascript application/x-javascript application/javascript text/xml application/xml application/xml+rss application/vnd.groove-tool-template application/octet-stream;

  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/global.d/globalblacklist.conf;
}

wooshop.com.tw

server {
    listen 80;

    server_name wooshop.com.tw;
    include conf.d/letsencrypt.inc;

    # Bad Bot Blocker
    include /etc/nginx/bots.d/blockbots.conf;
    include /etc/nginx/bots.d/ddos.conf;

    # Redirect all HTTP requests to HTTPS with a 301 Moved Permanently response.
    return 301 https://$host$request_uri;
}

server {
  listen 443 ssl http2;
  root /var/www/varnish2;
  server_name wooshop.com.tw;
  index index.html;
  resolver 172.17.0.1 valid=60s ipv6=off;
  set $backend varnish2;
  include conf.d/letsencrypt.inc;

  ssl_certificate /etc/letsencrypt/live/wooshop.com.tw/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/wooshop.com.tw/privkey.pem;

  ssl_session_timeout 1d;
  ssl_session_cache shared:SSL:50m;
  ssl_session_tickets off;

  # Diffie-Hellman parameter for DHE ciphersuites
  ssl_dhparam /d2c/ssl/dhparam.pem;

  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-128-GCM-SHA256:TLS13-AES-128-CCM-8-SHA256:TLS13-AES-128-CCM-SHA256";
  ssl_prefer_server_ciphers on;
  ssl_ecdh_curve secp521r1:secp384r1:prime256v1;

  # HSTS (15768000 seconds = 6 months)
  add_header Strict-Transport-Security max-age=15768000;

  # OCSP Stapling ---
  # fetch OCSP records from URL in ssl_certificate and cache them
  ssl_stapling on;
  ssl_stapling_verify on;

    # Bad Bot Blocker
        include /etc/nginx/bots.d/blockbots.conf;
        include /etc/nginx/bots.d/ddos.conf;

  location ~ /\.git {

    deny all;
  }

  try_files $uri/index.html $uri @app;
  location @app {
    proxy_pass        http://$backend:80;
    proxy_redirect    off;
    proxy_set_header  Upgrade $http_upgrade;
    proxy_set_header  Connection "upgrade";
    proxy_set_header  Host $host;
    proxy_set_header  X-Real-IP $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
  }
}
robert1112 commented 6 years ago

Hi @mitchellkrogza It works. But when I ran from Mac, the reposes is curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1) . If I ran from Linux, the response is curl: (56) GnuTLS recv error (-110): The TLS connection was non-properly terminated.

Is it correct? Thank you so much.

robert1112 commented 6 years ago

I got A from https://www.ssllabs.com/ssltest/index.html Will (92) be a problem to bad bot blocker function? Thank you so much.

mitchellkrogza commented 6 years ago

I'll update those config files slightly in the morning for you. Also please post what you have in letsencrypt.inc for me

robert1112 commented 6 years ago

Hi @mitchellkrogza Thanks for prompt reply. Here is what I used finally and the info you requested.

root@web2:/etc/nginx# cat nginx.conf

user  nginx;
worker_processes auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
  worker_connections  1024;
}

http {
  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;
  #server_names_hash_bucket_size 64;

  log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';

  access_log  /var/log/nginx/access.log  main;

  sendfile        on;
  #tcp_nopush     on;

  keepalive_timeout  65;

  client_body_buffer_size 32K;
  client_max_body_size 10M;

  client_body_timeout 10s;
  client_header_buffer_size 1k;
  large_client_header_buffers 2 16k;
  client_header_timeout 5s;

  gzip on;
  gzip_static on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied expired no-cache no-store private auth;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json text/javascript application/x-javascript application/javascript text/xml application/xml application/xml+rss application/vnd.groove-tool-template application/octet-stream;

  include /etc/nginx/conf.d/*.conf;
  include /etc/nginx/global.d/botblocker-nginx-settings.conf;
  include /etc/nginx/global.d/globalblacklist.conf;
}

root@web2:/etc/nginx/conf.d# cat 5b49c36d5073c9370fcf7d60.conf

server {
  listen 80;

  server_name a.wooshop.com.tw;
  include conf.d/letsencrypt.inc;

  # Bad Bot Blocker
    include /etc/nginx/bots.d/blockbots.conf;
    include /etc/nginx/bots.d/ddos.conf;

  # Redirect all HTTP requests to HTTPS with a 301 Moved Permanently response.
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl http2;

  ssl_certificate /etc/letsencrypt/live/a.wooshop.com.tw/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/a.wooshop.com.tw/privkey.pem;

  ssl_session_timeout 1d;
  ssl_session_cache shared:SSL:50m;
  ssl_session_tickets off;

  # Diffie-Hellman parameter for DHE ciphersuites
  ssl_dhparam /d2c/ssl/dhparam.pem;

  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:TLS13-AES-256-GCM-SHA384:TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-128-GCM-SHA256:TLS13-AES-128-CCM-8-SHA256:TLS13-AES-128-CCM-SHA256";
  ssl_prefer_server_ciphers on;
  ssl_ecdh_curve secp521r1:secp384r1:prime256v1;

  # HSTS (15768000 seconds = 6 months)
  add_header Strict-Transport-Security max-age=15768000;

  # OCSP Stapling ---
  # fetch OCSP records from URL in ssl_certificate and cache them
  ssl_stapling on;
  ssl_stapling_verify on;

  root /var/www/varnish2;
  server_name a.wooshop.com.tw;
  index index.html;
  resolver 172.17.0.1 valid=60s ipv6=off;
  set $backend varnish2;
  include conf.d/letsencrypt.inc;
  # Bad Bot Blocker
        include /etc/nginx/bots.d/blockbots.conf;
        include /etc/nginx/bots.d/ddos.conf;
  location ~ /\.git {
    deny all;
  }

  try_files $uri/index.html $uri @app;
  location @app {
    proxy_pass        http://$backend:80;
    proxy_redirect    off;
    proxy_set_header  Upgrade $http_upgrade;
    proxy_set_header  Connection "upgrade";
    proxy_set_header  Host $host;
    proxy_set_header  X-Real-IP $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  X-Forwarded-Proto $scheme;
  }
}

root@web2:/etc/nginx/conf.d# cat letsencrypt.inc

location ^~ /.well-known/acme-challenge/ {
    default_type "text/plain";
    root  /etc/letsencrypt/www;
}

location = /.well-known/acme-challenge/ {
    return 404;
}

THANK YOU SO MUCH.

mitchellkrogza commented 6 years ago

Can you comment out these lines for now, don't suggest you use stapling until you know the consequences

ssl_stapling on; ssl_stapling_verify on;

robert1112 commented 6 years ago

Hi @mitchellkrogza I just comment out these 2 lines like this 

# OCSP Stapling ---
  # fetch OCSP records from URL in ssl_certificate and cache them
  #ssl_stapling on;
  #ssl_stapling_verify on;

and here is the output. Thank you.

curl -A "Xenu Link Sleuth/1.3.8" https://a.wooshop.com.tw
curl: (92) HTTP/2 stream 1 was not closed cleanly: PROTOCOL_ERROR (err 1)
robert1112 commented 6 years ago

Hi @mitchellkrogza It occurred to me that I have CloudFlare in between and gray clouded, so I enabled http proxy and run test again. Here is the output.

~ curl -A "Xenu Link Sleuth/1.3.8" https://a.wooshop.com.tw

<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>

<title>a.wooshop.com.tw | 520: Web server is returning an unknown error</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robots" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1,maximum-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />
<!--[if lt IE 9]><link rel="stylesheet" id='cf_styles-ie-css' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->
<style type="text/css">body{margin:0;padding:0}</style>

</head>
<body>
<div id="cf-wrapper">

    <div id="cf-error-details" class="cf-error-details-wrapper">
        <div class="cf-wrapper cf-error-overview">
            <h1>

              <span class="cf-error-type">Error</span>
              <span class="cf-error-code">520</span>
              <small class="heading-ray-id">Ray ID: 442be22978a94656 &bull; 2018-07-31 00:17:41 UTC</small>
            </h1>
            <h2 class="cf-subheadline">Web server is returning an unknown error</h2>
        </div><!-- /.error-overview -->

        <div class="cf-section cf-highlight cf-status-display">
            <div class="cf-wrapper">
                <div class="cf-columns cols-3">

<div id="cf-browser-status" class="cf-column cf-status-item cf-browser-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-browser"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">You</span>
  <h3 class="cf-status-name">Browser</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-cloudflare-status" class="cf-column cf-status-item cf-cloudflare-status ">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-cloud"></i>
    <i class="cf-icon-status cf-icon-ok"></i>
  </div>
  <span class="cf-status-desc">Taipei</span>
  <h3 class="cf-status-name">Cloudflare</h3>
  <span class="cf-status-label">Working</span>
</div>

<div id="cf-host-status" class="cf-column cf-status-item cf-host-status cf-error-source">
  <div class="cf-icon-error-container">
    <i class="cf-icon cf-icon-server"></i>
    <i class="cf-icon-status cf-icon-error"></i>
  </div>
  <span class="cf-status-desc">a.wooshop.com.tw</span>
  <h3 class="cf-status-name">Host</h3>
  <span class="cf-status-label">Error</span>
</div>

                </div>

            </div>
        </div><!-- /.status-display -->

        <div class="cf-section cf-wrapper">
            <div class="cf-columns two">
                <div class="cf-column">
                    <h2>What happened?</h2>
                    <p>There is an unknown connection issue between Cloudflare and the origin web server. As a result, the web page can not be displayed.</p>
                </div>

                <div class="cf-column">
                    <h2>What can I do?</h2>
                          <h5>If you are a visitor of this website:</h5>
      <p>Please try again in a few minutes.</p>

      <h5>If you are the owner of this website:</h5>
      <p><span>There is an issue between Cloudflare's cache and your origin web server. Cloudflare monitors for these errors and automatically investigates the cause. To help support the investigation, you can pull the corresponding error log from your web server and submit it our support team.  Please include the Ray ID (which is at the bottom of this error page).</span> <a href="https://support.cloudflare.com/hc/en-us/articles/200171936-Error-520">Additional troubleshooting resources</a>.</p>
                </div>
            </div>

        </div><!-- /.section -->

        <div class="cf-error-footer cf-wrapper">
  <p>
    <span class="cf-footer-item">Cloudflare Ray ID: <strong>442be22978a94656</strong></span>
    <span class="cf-footer-separator">&bull;</span>
    <span class="cf-footer-item"><span>Your IP</span>: 121.254.111.144</span>
    <span class="cf-footer-separator">&bull;</span>
    <span class="cf-footer-item"><span>Performance &amp; security by</span> <a href="https://www.cloudflare.com/5xx-error-landing?utm_source=error_footer" id="brand_link" target="_blank">Cloudflare</a></span>

  </p>
</div><!-- /.error-footer -->

    </div><!-- /#cf-error-details -->
</div><!-- /#cf-wrapper -->
</body>
</html>

curl -I https://a.wooshop.com.tw -e http://100dollars-seo.com

HTTP/2 520
date: Tue, 31 Jul 2018 00:25:10 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=dfb7e864a05a29e610e84fe33ffb32ee51532996710; expires=Wed, 31-Jul-19 00:25:10 GMT; path=/; domain=.wooshop.com.tw; HttpOnly
expires: Thu, 01 Jan 1970 00:00:01 GMT
cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
pragma: no-cache
x-frame-options: SAMEORIGIN
set-cookie: cf_use_ob=0; expires=Tue, 31-Jul-18 00:25:40 GMT; path=/
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 442bed2149d845ae-TPE

curl -A "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" https://a.wooshop.com.tw It will show the content of the html. I think that is it, right? THANK YOU SO MUCH.

mitchellkrogza commented 6 years ago

@robert1112 it seems like it's working. Cloudflare of course gives a different error but if you check your local logs you should see 444 errors showing.

With my User-Agent set to XENU in Chrome

nginx

With my User-Agent set to Chrome (Default) in Chrome

nginx2

👍

mitchellkrogza commented 6 years ago

Oh @robert1112 sorry you can re-enable these settings, it was late at night and I was thinking of something else entirely.

ssl_stapling on;
ssl_stapling_verify on;
robert1112 commented 6 years ago

Hi @mitchellkrogza OK I will uncomment those. After 4 days of work, I am glad I make it. 👍 THANK YOU SO MUCH FOR YOUR SUPPORT.

mitchellkrogza commented 6 years ago

So glad it's working now @robert1112 patience is the key to learning and well done on persevering.

robert1112 commented 6 years ago

Hi @mitchellkrogza Yes, patience is the key. Your support is the other key. 👍 I tried with this but seems not to be blocked. I don't know why. https://builtwith.com/a.wooshop.com.tw

robert1112 commented 6 years ago

Some minor thoughts here. If these lines are added in my my config file, do I still need to run sudo ./setup-ngxblocker -x -v /etc/nginx/conf.d -e conf

include /etc/nginx/global.d/botblocker-nginx-settings.conf;
include /etc/nginx/global.d/globalblacklist.conf;
# Bad Bot Blocker
    include /etc/nginx/bots.d/blockbots.conf;
    include /etc/nginx/bots.d/ddos.conf;
mitchellkrogza commented 6 years ago

@robert1112 builtwith.com changes IP addresses all the time, I block them only by IP as they hide their user agent. I will need to revisit them and do some tests to get their latest IP addresses.

mitchellkrogza commented 6 years ago

If you only have one or two sites, just do your includes manually it takes all of 5 seconds.

robert1112 commented 6 years ago

@mitchellkrogza Thanks for the clarification. I think we will not be able to list all Builwith.com ips since they will change from time to time. One more thing, I dont have access.log or error.log file on my host in docker environment and as such, I cannot use Fail2Ban to monitor logs. What is the impact on full function of your tool? Aggressive bot cannot be restricted? Thank you.

mitchellkrogza commented 6 years ago

Anything can be blocked out assuming you know their IP's and/or User-Agent strings. That's why having logs to monitor daily is so important.

robert1112 commented 6 years ago

Hi @mitchellkrogza

It is frustrated but I cannot have logs in host without going out of my way to set up API service first. (that is said by the PaaS) They centralized all logs on their server, I think.

Thank you so much for the support all the way.

mitchellkrogza commented 6 years ago

and What happens if you specify the log file locations in the vhost config ??

server {
  listen 80;

  server_name a.wooshop.com.tw;
  include conf.d/letsencrypt.inc;

    access_log /var/log/nginx/a.wooshop.com.tw-access.log;
    error_log /var/log/nginx/a.wooshop.com.tw-error.log;

    .....
    .....
    .....

}

server {
  listen 443 ssl http2;

  ssl_certificate /etc/letsencrypt/live/a.wooshop.com.tw/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/a.wooshop.com.tw/privkey.pem;

    access_log /var/log/nginx/a.wooshop.com.tw-access.log;
    error_log /var/log/nginx/a.wooshop.com.tw-error.log;

    .....
    .....
    .....
    .....
}
robert1112 commented 6 years ago

WOWx2, it works. It works. So I can monitor this. BIG THANKS, @mitchellkrogza