Setup is successful but bots aren't blocked

[ghost]

Hi, I used the manual installation method because my server directories are totally different as I use Webuzo to manage my server. After following the steps, the nginx -t command shows successful but the bots aren't blocked when I use the test method given in the tutorial.

[mitchellkrogza]

Please post your nginx.conf and your vhost.conf here for me

[ghost]

This file is the vhost file, which is basically to prevent webuzo from overriding changes as the vhost file is updated by webuzo. Changes made in this file are like changed made in the actual file. Just they won't be overriden in updates. https://pastebin.com/F0vVKkmv

This is the vhost file generated by webuzo https://pastebin.com/R3nH9vWR

This is the nginx.conf file https://pastebin.com/1gmcfziY

[mitchellkrogza]

@faizan47 with your first pastebin link, is that this file you are referring to? include /usr/local/apps/nginx/etc/conf.d/common;

[mitchellkrogza]

You've also done your includes wrong in nginx.conf.

You have this in your nginx.conf

http {
............
include /usr/local/apps/nginx/etc/conf.d/*.conf;
include /usr/local/apps/nginx/etc/bots.d/blockbots.conf;
include /usr/local/apps/nginx/etc/bots.d/ddos.conf;
...........
}

That is wrong, the only include in nginx.conf is this

http {
.............
include /usr/local/apps/nginx/etc/conf.d/*.conf;
.............
}

These two includes below MUST be inside a server {} block not an http {} block.

server {
.............
include /usr/local/apps/nginx/etc/bots.d/blockbots.conf;
include /usr/local/apps/nginx/etc/bots.d/ddos.conf;
............
}

[mitchellkrogza]

So your nginx.conf will look like this.

#user  nobody;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

pid     /usr/local/apps/nginx/var/log/nginx.pid;

events {
    worker_connections  1024;
}

error_log /usr/local/apps/nginx/var/log/error_log debug;

http {
    include    mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /usr/local/apps/nginx/var/log/web.access.log  main;

    sendfile        on;
    #tcp_nopush  on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    client_max_body_size 200M;

    # If your domain names are long, increase this parameter.
    server_names_hash_bucket_size 64;

    # To hide the version number in headers
    server_tokens off;

    include /usr/local/apps/nginx/etc/conf.d/*.conf;
}

Your custom file will look like this.

# Bad Bot Blocker
include /usr/local/apps/nginx/etc/bots.d/blockbots.conf;
include /usr/local/apps/nginx/etc/bots.d/ddos.conf;

# Redirect server error pages to the static pages
error_page 500 502 504  /500.html;
location = /500.html {
    root    /usr/local/apps/nginx/etc/conf.d;
    internal;
}

error_page  404         /404.html;
location = /404.html {
    root    /usr/local/apps/nginx/etc/conf.d;
    internal;
}

error_page  497         /497.html;
location = /497.html {
    root    /usr/local/apps/nginx/etc/conf.d;
    internal;
}

error_page  555 /555.html;
location = /555.html {
    root    /usr/local/apps/nginx/etc/conf.d;
    internal;
}

location @maintenance {
    root    /usr/local/apps/nginx/etc/conf.d;
    rewrite ^(.*)$ /noindex.html break;
}

# Pass the INDEX.PHP script to FastCGI server listening on 127.0.0.1:9178
location = / {
    fastcgi_read_timeout 3600;
    try_files $uri /index.php /index.html @maintenance;
    fastcgi_pass    unix:/usr/local/apps/php70/var/php70_9001.sock;
    fastcgi_param   SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param   SCRIPT_NAME     $fastcgi_script_name;
    include         fastcgi_params;
}
# Fix permalinks WordPress
location / {
try_files $uri $uri/ /index.php?$args;
}
# Security
location ~* /(?:uploads|files)/.*.php$ {
    deny all;
}
# WooCommerce No cache
if ($request_uri ~* "/(cart|checkout|my-account)/*$") {
    set $skip_cache 1;
}

## block any attempted XML-RPC requests
location = /xmlrpc.php {
    deny all;
}
# For userdir files
location ~* ^/~(.+?)(/.*\.php)$ {

    alias /home/$1/public_html$2;
    fastcgi_pass    unix:/usr/local/apps/php70/var/php70_9001.sock;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME $request_filename;
}

# For userdir files
location ~ ^/~(.+?)(/.*)?$ {
    alias /home/$1/public_html$2;
    index  index.html index.htm index.php;
    autoindex on;
}

# To catch index.php by default
location ~ (index.php|/)$ {
    fastcgi_read_timeout 3600;
    try_files  $uri $uri/index.php $uri/index.html;
    fastcgi_pass    unix:/usr/local/apps/php70/var/php70_9001.sock;
    fastcgi_param   SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param   SCRIPT_NAME     $fastcgi_script_name;
    include         fastcgi_params;
}

# Pass the regular PHP scripts to FastCGI server listening on 127.0.0.1:9179
location ~ \.php$ {
    fastcgi_read_timeout 3600;
    try_files $uri =404;
    fastcgi_pass    unix:/usr/local/apps/php70/var/php70_9001.sock;
    fastcgi_index   index.php;
    fastcgi_param   SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param   SCRIPT_NAME     $fastcgi_script_name;
    include     fastcgi_params;
}

location ~ \.pl|cgi$ {
    try_files $uri =404;
    gzip off;
    fastcgi_pass    127.0.0.1:8999;
    fastcgi_index   index.cgi;
    fastcgi_param   SCRIPT_FILENAME $document_root$fastcgi_script_name;
    fastcgi_param   SCRIPT_NAME     $fastcgi_script_name;
    include         fastcgi_params;
}

[ghost]

Thanks so much for your time and help. My server was stuck and hence everything got messed up. I will check and update you as I will have to install the software(webuzo) again. Thanks for your patience.

[mitchellkrogza]

That's what I'm here for 👍 let me know how it goes.

[ghost]

Hi. I have tried it. `curl -A "Xenu Link Sleuth/1.3.8" https://quvor.com

curl -A "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)" https://quvor.com

curl -I https://quvor.com -e http://100dollars-seo.com

curl -I https://quvor.com -e http://zx6.ru` These commands show me an error. I hope that's working. Also, I had to replace the correct directory of my server in globalblacklist.conf to make it work. Will this be overriden in updates? And will SSH brute force attacks be blocked by this? Thanks!

[mitchellkrogza]

I tested from outside and it's not blocking anything. Are you busy making any changes right now?

You should use the update-ngxblocker script to update for you as it takes care of automatically modifying the include locations inside globalblacklist.conf.

Where are your bots.d and conf.d folders located? Let me know and I will show you how to use the update-ngxblocker to do it for you.

[mitchellkrogza]

If I test from outside I get

curl -I https://quvor.com -e http://zx6.ru
HTTP/1.1 200 OK
Date: Sun, 10 Sep 2017 13:18:51 GMT
Server: Apache
X-Powered-By: PHP/7.0.21
Link: <https://quvor.com/wp-json/>; rel="https://api.w.org/"
Vary: Accept-Encoding,User-Agent
Content-Type: text/html; charset=UTF-8

It should give you this.

curl -I https://ubuntu101.co.za -e http://zx6.ru
curl: (52) Empty reply from server

[ghost]

Actually I had my site on my new server. But the I migrated. So, I current domain is setup1.tk and not quvor.com(it's on a shared host for now). I had that on this server in the past. Now it's on different server. Sorry about that. And yes, I was making changes. Sorry about it. /usr/local/apps/nginx/etc/(this is my actual directory which I replaced in the code) with /etc/nginx/ This is what I get now with the command curl -I http://setup1.tk -e http://zx6.ru HTTP/1.1 520 Origin Error Date: Sun, 10 Sep 2017 13:30:05 GMT Content-Type: text/html; charset=UTF-8 Connection: keep-alive Set-Cookie: __cfduid=db2fc9301b963fdc3ea34c730122332ba1505050205; expires=Mon, 10-Sep-18 13:30:05 GMT; path=/; domain=.setup1.tk; HttpOnly Expires: Thu, 01 Jan 1970 00:00:01 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache X-Frame-Options: SAMEORIGIN Server: cloudflare-nginx CF-RAY: 39c2bd6b333b58a3-DFW

[mitchellkrogza]

Also not working correctly.

curl -I http://setup1.tk -e http://zx6.ru
HTTP/1.1 520 Origin Error
Expires: Thu, 01 Jan 1970 00:00:01 GMT
Content-Type: text/html; charset=UTF-8
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Date: Sun, 10 Sep 2017 13:36:50 GMT
Set-Cookie: __cfduid=dedf67a5237b5f1c56d69dff5fda0d85f1505050610; expires=Mon, 10-Sep-18 13:36:50 GMT; path=/; domain=.setup1.tk; HttpOnly
Pragma: no-cache
X-Frame-Options: SAMEORIGIN
Server: cloudflare-nginx
CF-RAY: 39c2c748b55a15f2-JNB
Connection: Close

Should be

curl -I http://setup1.tk -e http://zx6.ru
curl: (52) Empty reply from server

[mitchellkrogza]

You should use update-ngxblocker as follows

sudo /usr/sbin/update-ngxblocker -c /usr/local/apps/nginx/etc/conf.d -b /usr/local/apps/nginx/etc/bots.d -e yourname@youremail.com

This will make sure it downloads latest list and fixes the include locations inside globalblacklist.conf for you without you having to go and fiddle manually.

[mitchellkrogza]

The 520 Origin Error is being caused by cloudflare. See: https://support.cloudflare.com/hc/en-us/articles/200171936-Error-520-Web-server-is-returning-an-unknown-error

[ghost]

Thanks for the command. What should be done for cloudflare causing the issue? Will disabling it temporarily to check the bot blocked work? And after this if I enable it again, it will work?

[mitchellkrogza]

Yes disable cloudflare and test to make sure it is working.

[ghost]

Okay. I will update. DNS propogation will take some time. I had changed the domain quvor for branding purposes. Since setup1.tk is just like dummy. I'm just testing the vps performance on different setup. Thanks :) will update soon

[ghost]

Hi. I disabled cloudflare. I am getting empty respond now. Please confirm if it's working. Also, will this stop working with cloudflare?

On 10-Sep-2017 7:32 PM, "Mitchell Krog" notifications@github.com wrote:

Yes disable cloudflare and test to make sure it is working.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/issues/92#issuecomment-328344673, or mute the thread https://github.com/notifications/unsubscribe-auth/AeUokjpakG_5cTzG224mboNI11t2HQyzks5sg-wGgaJpZM4PR82Z .

[ghost]

Also, does this script block SSH access from bots and ips too? Some one is trying to brute force my server.

On 10-Sep-2017 7:54 PM, "Faizan Shaikh" fzn9898@gmail.com wrote:

Hi. I disabled cloudflare. I am getting empty respond now. Please confirm if it's working. Also, will this stop working with cloudflare?

On 10-Sep-2017 7:32 PM, "Mitchell Krog" notifications@github.com wrote:

Yes disable cloudflare and test to make sure it is working.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/issues/92#issuecomment-328344673, or mute the thread https://github.com/notifications/unsubscribe-auth/AeUokjpakG_5cTzG224mboNI11t2HQyzks5sg-wGgaJpZM4PR82Z .

[mitchellkrogza]

curl -I http://setup1.tk -e http://zx6.ru
curl: (52) Empty reply from server

👍

Nope the blocker won't help you with attacks on SSH and other services, for that you need Fail2Ban. Don't panic though, it may look bad to you, but I have 6 servers and they are all under constant SSH attack 24/7/365 ..... Fail2Ban will help you block them and my custom Perma-Banning Filter will also help you block them for extended periods.

[ghost]

Thanks so much for your quick replies. I will try fail2ban with your script. Thanks one more time.

On 10-Sep-2017 8:00 PM, "Mitchell Krog" notifications@github.com wrote:

curl -I http://setup1.tk -e http://zx6.ru curl: (52) Empty reply from server

👍

Nope the blocker won't help you with attacks on SSH and other services, for that you need Fail2Ban. Don't panic though, it may look bad to you, but I have 6 servers and they are all under constant SSH attack 24/7/365 ..... Fail2Ban will help you block them and my custom Perma-Banning Filter https://github.com/mitchellkrogza/Fail2Ban-Blacklist-JAIL-for-Repeat-Offenders-with-Perma-Extended-Banning will also help you block them for extended periods.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/issues/92#issuecomment-328346289, or mute the thread https://github.com/notifications/unsubscribe-auth/AeUokndsVvqV-sAhTSYs4Ipgi7Bkh2UJks5sg_J1gaJpZM4PR82Z .

[itoffshore]

a better solution than fail2ban is fwknop to stealth ssh completely

I use fwknop with 4096 bit keys

[itoffshore]

a quick way to stop ssh being bruteforced is to NOT put it on port 22 & connect to your server with:

ssh -p xxxx user@server.com

with xxxx being the non standard port number

you could also only use a vpn to connect over ssh & limit access with iptables to the vpn ip address.

[mitchellkrogza]

@faizan47 can I close this issue now? All working ok ??

[ghost]

Hi sorry was a bit busy. Yes. Thanks a ton. I will be free from most of the spam now. Thanks to GitHub and developers like you :)

[mitchellkrogza]

:+1: