Feature Suggestion / Integration: Apache Ultimate Bad Bot Blocker

[abclution]

Super cool, dynamic list of a$$hole bots blocker + fail2ban integration as well. https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker

Just checking this out today as a bunch of my sites are continually hammered by bots.

From what I can tell the only thing really needed is the automatic insertion into the apache virtual hosts + fail2ban filter/jail. Just starting the setup now, so I may have more feedback afterwards. Looks like I can include some of it into the settings / templates.

@chris001 Have you seen / tried this?

[chris001]

@abclution I haven't seen or tried it yet, but it looks very interesting, however I wonder if it's so strong it might block Google bot from crawling your site and if so then that could possibly damage your Google SEO ranking? Would be great to try out the Nginx version also: https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker

[abclution]

@chris001 By default it allows "good bots" and blocklists and allowlists are fully customizable.. Sane defaults such as google / bing bots are allowed.

Can allow / block via agent, ip I managed to implement it pretty easily.

Server load dropped 30% immediately. Still testing but so far I am happy about it.

[abclution]

Here is a partial walkthrough to enable this globally for all sites on a Virtualmin Install using Apache 2.4+ YMMV I used instructions from here and did the process manually. https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/tree/master/Apache_2.4

mkdir -p /etc/apache2/custom.d

wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/globalblacklist.conf -O /etc/apache2/custom.d/globalblacklist.conf

wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/blacklist-user-agents.conf -O /etc/apache2/custom.d/blacklist-user-agents.conf

NOTE: On the Apache Blocker if you want to over-ride any of the whitelisted bots you can add them to this include file and the previously whitelisted bots in the blocker will be over-ridden by this include file

wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/bad-referrer-words.conf -O /etc/apache2/custom.d/bad-referrer-words.conf wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/blacklist-ips.conf -O /etc/apache2/custom.d/blacklist-ips.conf

wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/whitelist-ips.conf -O /etc/apache2/custom.d/whitelist-ips.conf wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/whitelist-domains.conf -O /etc/apache2/custom.d/whitelist-domains.conf

ENABLE GLOBALLY ON ALL VIRTUAL HOSTS

Create / edit file /etc/apache2/site-availiable/zzzzzz.blocklist.conf !IMPORTANT, the config zzzzzz.blocklist.conf must be named in a way that it is loaded LAST OF ALL CONFIGS. Thats how it applies to ALL VHOSTS, hence zzzzzz*

# ------------------/etc/apache2/site-availiable/zzzzzz.blocklist.conf-------------------------
# ######################################
# GLOBAL! deny bad bots and IP addresses
# ######################################
#
# should be set after <VirtualHost>s see https://httpd.apache.org/docs/2.4/sections.html#merging
<Location "/">
        # AND-combine with preceding configuration sections
        AuthMerging And
        # include black list
        Include custom.d/globalblacklist.conf
</Location>
# -------------------------------------------

!IMPORTANT In step 2 of their instructions here: https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/tree/master/Apache_2.4 "Use nano, vim or any other text editor to edit both whitelist-ips.conf and whitelist-domains.conf to include all your own domain names and IP addresses that you want to specifically whitelist from the blocker script." They mention to whitelist all you domains and ips. To be honest I didn't quite understand WHY it would say that and I didn't do it myself as it didnt seem neccessary.

!IMPORTANT IPV6 IP Whitelist entries for ipv6 should NOT be encapsulated by [IPV6] brackets, just the plain IP worked.

ENABLE THE APACHE CONFIG / Link the config from available to enabled ln -s /etc/apache2/sites-available/zzzzzz.blocklist.conf /etc/apache2/sites-enabled/zzzzzz.blocklist.conf

TEST CONFIG apache2ctl configtest

RESTART APACHE2 service apache2 restart

TEST BLOCKER IS WORKING

SHOULD PASS curl -A "googlebot" http://yourdomain.com Should respond with 200 OK

SHOULD FAIL - BAD USER AGENT curl -A "80legs" http://yourdomain.com curl -A "masscan" http://yourdomain.com Should respond with 403 Forbidden

SHOULD FAIL - BAD REFFERER curl -I http://yourdomain.com -e http://100dollars-seo.com curl -I http://yourdomain.com -e http://zx6.ru Should respond with 403 Forbidden

[chris001]

We should start by creating an actual install/test script (for apache2 and nginx) in bash shell script code, and share it here. The step by step you posted above is a good start, it just needs to test for errors and respond accordingly. Then, it'd be easier for the maintainers to give it a try and see if they'll add it to the official Virtualmin package as a default enabled HTTP web server protective security service to block nuisance bots and reduce server load, improve website speed and responsiveness for our actual users. Most huge successful web sites (including github itself) block bad/nuisance/hacker bots, in some way or another. It's kind of a lost opportunity to recover server processing power NOT to block them. Consider those bad bots as a constantly there, relentless DDoS attack. Nobody's got time for that!

[abclution]

Oh I forgot to mention, it comes with a bash shell script to install, + updater script + few other things. I just did it manually cause I was unsure of the process and wanted to fully understand the underpinnings but the shell script that is included seems clear and clean (and compatible). Only the last step in creating the zzzzzzzzzzz.blocker.conf is "batteries not included"

Out of time for the moment to muck with it more but hope to revisit it soon, has made a HUGE difference, and I haven't even yet done the fail2ban integration. Bots hammering wordpress sites vs getting 403 really reduces the load. Highly reccomended.

[chris001]

Test installing it on nginx.

First problem: The Nginx I'm running says it's full nginx 1.10 but it's really nginx-light because it was compiled without a required module ngx_http_limit_req http://nginx.org/en/docs/http/ngx_http_limit_req_module.html . This bad bot blocker requires nginx with that module compiled in:

# nginx -t
nginx: [emerg] unknown directive "limit_req_zone" in /etc/nginx/conf.d/botblocker-nginx-settings.conf:20
nginx: configuration file /etc/nginx/nginx.conf test failed

Second problem (indirectly related): While trying to update to a new nginx full version with this required limit_req module compiled in, it also wants to update virtualmin, however, the "Setting up" stage of the Virtualmin install fails with errors of file not found on ancient outdated missing letsencrypt cert files?

Setting up webmin-virtual-server (6.16.gpl) ...

Use of uninitialized value $mode in string ne at /usr/share/webmin/virtualmin-nginx/virtual_feature.pl line 956.
Use of uninitialized value $extras[0] in join or string at /usr/share/webmin/virtualmin-nginx/virtual_feature.pl line 944.
Use of uninitialized value $extras[1] in join or string at /usr/share/webmin/virtualmin-nginx/virtual_feature.pl line 944.
Use of uninitialized value $mode in string ne at /usr/share/webmin/virtualmin-nginx/virtual_feature.pl line 956.
Use of uninitialized value $mode in string ne at /usr/share/webmin/virtualmin-nginx/virtual_feature.pl line 956.
virtual-server/postinstall.pl failed : virtual-server::module_install failed : Failed to open /etc/letsencrypt/live/mydomain.com/../../archive/mydomain.com/cert9.pem.webmintmp.7980 : Permission denied

[chris001]

Virtualmin and Nginx apt package upgrade errors resolved. Nginx Ultimate Bad Bot Blocker installed/running/testing on nginx 1.16.0 on Debian right now.

[abclution]

I don't run Nginx on my virtualmin servers, so I didn't get to test it. But I DID see the Nginx version of ABBB has throttling control support on nginx (as it is built into nginx, which is compelling by itself.)

I am guessing that is the module that gave you a hard time, and whether its worth getting different packages or disabling the throttling, or whatever the solution you came up with..

[chris001]

Upgraded from nginx-light package to the full nginx, which is compiled with the nginx HTTP module limit_req, to test this UBBB.

[swelljoe]

This isn't something we'd want to ship in a default installation. Too complicated, prone to breakage, and only really useful for a subset of users. It'd need to be a third-party module, I think. Though a brief scan through the docs makes it look like you could quite easily just add a couple/few lines to your Apache Server Template.