search

mitchellkrogza / apache-ultimate-bad-bot-blocker

Apache Block Bad Bots, (Referer) Spam Referrer Blocker, Vulnerability Scanners, Malware, Adware, Ransomware, Malicious Sites, Wordpress Theme Detectors and Fail2Ban Jail for Repeat Offenders
Other
807 stars 175 forks source link

Not Apache 2.4 compatible without mod_compat? #32

Closed BerndHelber closed 7 years ago

BerndHelber commented 7 years ago

Hi i was poking around to get your wonderful toolset up and running on SLES12/OpenSuSE Leap 42.2 I had to face that it would not run without mod_compat.

So i started this morning replacing the Apache 2.4 commands.

sed -i -e s/Allow from/Require ip/g' globalblacklist.con sed -i -e 's/deny from/Require not ip/g' globalblacklist.conf sed -i -e 's/deny from/Require not ip/g' .conf sed -i -e 's/Allow from/Require ip/g' .conf

Might be worth a shot, to implement this for a new release?

Kind regards.

mitchellkrogza commented 7 years ago

Thanks @BerndHelber for the heads up. I'm figuring I'm going to have to have 2 separate versions for older Apache 2.2 and newer Apache 2.4.

Where

# START GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
Allow from 108.177.8.0/21
Allow from 108.177.96.0/19
Allow from 172.217.0.0/19
Allow from 173.194.0.0/16

Becomes

# START GOOGLE IP RANGES ### DO NOT EDIT THIS LINE AT ALL ###
Require ip 108.177.8.0/21
Require ip 108.177.96.0/19
Require ip 172.217.0.0/19
Require ip 173.194.0.0/16

And

# START WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###
deny from 104.197.51.76
deny from 108.167.189.81
deny from 109.73.225.87
deny from 13.68.211.181

Becomes

# START WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###
Require not ip 104.197.51.76
Require not ip 108.167.189.81
Require not ip 109.73.225.87
Require not ip 13.68.211.181

I'll have a look into this for the next version release.

mitchellkrogza commented 7 years ago

Just reading the docs at: https://httpd.apache.org/docs/2.4/howto/access.html

Wondering if sections for bad referrers like this

# START BAD REFERERS ### DO NOT EDIT THIS LINE AT ALL ###
SetEnvIfNoCase Referer ~*000free\.us spam_ref=1
SetEnvIfNoCase Referer ~*007angels\.com spam_ref=1
SetEnvIfNoCase Referer ~*00author\.com spam_ref=1

Might have to change to

# START BAD REFERERS ### DO NOT EDIT THIS LINE AT ALL ###
Require not host 000free.us
Require not host 007angels.com
Require not host 00author.com

???

BerndHelber commented 7 years ago

Hi

first.. Syntax Test on the Existing Confifg and Rules

grep SetEnvIfNoCase *.conf apachectl configtest Syntax OK

Apache is stopping and starting properly.

For validation, i made a copy cp globalblacklist.conf globalblacklist.backup

Started Replaceing the SetEnvIfNoCase Rules

sed -i -e 's/SetEnvIfNoCase Referer/Require not host /g' globalblacklist.conf

apachectl configtest

AH00526: Syntax error on line 774 of /etc/apache2/custom.d/globalblacklist.conf: negative Require directive has no effect in directive /etc/apache2/custom.d #

So it seemed if the Required not host Rule should be replacing the SetEnvIF Rule we have notation Bug Line 774 773 # START BAD REFERERS ### DO NOT EDIT THIS LINE AT ALL ### 774 Require not host ~000free.us spam_ref=1 775 Require not host ~007angels.com spam_ref=1 776 Require not host ~00author.com spam_ref=1 777 Require not host ~00go.com spam_ref=1 I If i compare what i did with the Apache Docs it seems, that Apache has a problem with the Tilde.

I tried to replace the tilde baut i assume that wasn't the issue 774 Require not host *000free.us spam_ref=1

In case the Mesage is the same as before. :-)

mitchellkrogza commented 7 years ago

@BerndHelber yip I am running into the same problems, been testing this morning and also getting negative Require directive has no effect in directive busy looking into all the ramifications of mod_access_compat being deprecated soon and how this blocker will have to change to keep up with Apache 2.4 and future versions. I've been reading all over but there's a lot of conflicting info out there. Posted a Q on stackoverflow and shortly on Apache users group for some clarification.

mitchellkrogza commented 7 years ago

@BerndHelber let me know if you figure it out. For now using mod_access_compat might be a temporary solution until we get to the bottom of this. Will see what feedback I get on my Q's :)

mitchellkrogza commented 7 years ago

I have confirmed mod_access_compat is required so going to have to get to the bottom of this

~# a2dismod access_compat
Module access_compat disabled.
To activate the new configuration, you need to run:
  service apache2 restart

~# service apache2 restart
Job for apache2.service failed because the control process exited with error code. See "systemctl status apache2.service" and "journalctl -xe" for details.

# a2enmod access_compat
Considering dependency authn_core for access_compat:
Module authn_core already enabled
Enabling module access_compat.
To activate the new configuration, you need to run:
  service apache2 restart

~# service apache2 restart

and then it reloads

BerndHelber commented 7 years ago

Great Work!

I'll give it a shot, when I'm back in office on wednesday and will provide you feedback.

Kind regards

mitchellkrogza commented 7 years ago

Thanks, I keep plundering on as I always do. From what I have read so far the new access method's are quite different from Apache 2.4. From what I can understand thus far is that the blocker for Apache 2.4+ should only specify explicit deny's and not do any any whitelisting of ips ranges like cloudflare, google and bing. Not sure I am too impressed with this change to Apache.

This is because of the new Require all granted instead of the previous Order allow,deny

I have to test this but from what I can understand it would have to look something like this.

<RequireAll>

# ALLOW ALL EXCEPT WHAT IS DENIED BELOW
# *************************************

Require all granted

# START WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###
Require not ip 104.197.51.76
Require not ip 108.167.189.81
Require not ip 109.73.225.87
Require not ip 13.68.211.181
Require not ip 142.4.218.201
Require not ip 149.56.33.22
Require not ip 158.69.187.171
Require not ip 158.69.26.58
Require not ip 162.13.185.20
Require not ip 173.237.189.235
Require not ip 173.255.210.133
Require not ip 185.45.14.186
Require not ip 192.163.217.239
Require not ip 192.185.4.40
Require not ip 192.95.29.139
Require not ip 192.99.17.79
Require not ip 198.27.69.229
Require not ip 198.58.124.46
Require not ip 199.241.28.124
Require not ip 212.71.238.108
Require not ip 37.247.121.179
Require not ip 37.60.253.215
Require not ip 45.63.68.250
Require not ip 45.79.139.191
Require not ip 50.116.84.148
Require not ip 52.87.112.125
Require not ip 66.96.183.60
Require not ip 89.36.223.188
# END WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###

# START NIBBLER ### DO NOT EDIT THIS LINE AT ALL ###
Require not ip 52.201.238.175
Require not ip 52.90.20.216
Require not ip 54.161.247.146
Require not ip 54.211.214.177
Require not ip 54.227.194.252
Require not ip 54.242.239.179
Require not ip 54.242.250.203
# END NIBBLER ### DO NOT EDIT THIS LINE AT ALL ###

# ****************************
# Known Bad IP's and IP Ranges
# *************************************************
# Blacklist IP addresses and IP Ranges Customizable
# *************************************************

Include "/etc/apache2/custom.d/blacklist-ips.conf"

# BLOCK OUT BAD BOTS AND REFERRERS
# ********************************

Require not env bad_bot
Require not env spam_ref

</RequireAll>
mitchellkrogza commented 7 years ago

Actually on a second glance at someone's code example, whitelisting ranges and domains may still be possible with something like this. I will test this and see what happens 😬 

<RequireAny>
    <RequireAll>

# ALLOW ALL EXCEPT WHAT IS DENIED BELOW
# *************************************

Require all granted

# START WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###
Require not ip 104.197.51.76
Require not ip 108.167.189.81
Require not ip 109.73.225.87
Require not ip 13.68.211.181
Require not ip 142.4.218.201
Require not ip 149.56.33.22
Require not ip 158.69.187.171
Require not ip 158.69.26.58
Require not ip 162.13.185.20
Require not ip 173.237.189.235
Require not ip 173.255.210.133
Require not ip 185.45.14.186
Require not ip 192.163.217.239
Require not ip 192.185.4.40
Require not ip 192.95.29.139
Require not ip 192.99.17.79
Require not ip 198.27.69.229
Require not ip 198.58.124.46
Require not ip 199.241.28.124
Require not ip 212.71.238.108
Require not ip 37.247.121.179
Require not ip 37.60.253.215
Require not ip 45.63.68.250
Require not ip 45.79.139.191
Require not ip 50.116.84.148
Require not ip 52.87.112.125
Require not ip 66.96.183.60
Require not ip 89.36.223.188
# END WP THEME DETECTORS ### DO NOT EDIT THIS LINE AT ALL ###

# START NIBBLER ### DO NOT EDIT THIS LINE AT ALL ###
Require not ip 52.201.238.175
Require not ip 52.90.20.216
Require not ip 54.161.247.146
Require not ip 54.211.214.177
Require not ip 54.227.194.252
Require not ip 54.242.239.179
Require not ip 54.242.250.203
# END NIBBLER ### DO NOT EDIT THIS LINE AT ALL ###

# ****************************
# Known Bad IP's and IP Ranges
# *************************************************
# Blacklist IP addresses and IP Ranges Customizable
# *************************************************

Include /etc/apache2/custom.d/blacklist-ips.conf

# BLOCK OUT BAD BOTS AND REFERRERS
# ********************************

Require not env bad_bot
Require not env pam_ref

    </RequireAll>

    <RequireAny>
    # have whitelisted ip ranges and domains here
    </RequireAny>

</RequireAny>
mitchellkrogza commented 7 years ago

This requires a lot more testing but so far this block layout seems to be the correct way to go. I will test this more and push out a 2.4 version this week for you to test.

<Directory /var/www/html>
<RequireAny>
    <RequireAll>

Require all granted
Require not ip 104.197.51.76
Require not ip 54.242.250.203
Require not env bad_bot
Require not env spam_ref

    </RequireAll>

    <RequireAny>
    Require env good_ref
    Require env good_bot
    </RequireAny>

</RequireAny>
</Directory>
mitchellkrogza commented 7 years ago

@BerndHelber try this new test unit and let me know - https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/tree/master/Apache_2.4/custom.d

I have tested it and so far it seems to be working perfectly. I have stripped this version down removing all commenting and also commenting out any of the custom include files for now.

Please test and report back.

mitchellkrogza commented 7 years ago

@BerndHelber any luck trying out the new test unit ???

BerndHelber commented 7 years ago

Hello Mitchell. Was rather busy the last few days.

Tomorrow morning i will give it a shot. Sorry to let you wait.

BerndHelber commented 7 years ago

Ok....

:/etc/apache2/custom.d # a2dismod access_compat "access_compat" not present

Mod Compat not present in SuSE/SLES 12 seems to be CentOS/RHEL only feature.

cd /etc/apache2 tar -cf securecustomsettings.tar custom.d/

cd custom.d/ rm *

wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/blacklist-ips.conf

wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/blacklist-user-agents.conf

wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/globalblacklist.conf

wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/whitelist-domains.conf

wget https://raw.githubusercontent.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/master/Apache_2.4/custom.d/whitelist-ips.conf

Apache Config test

apachectl configtest apachectl restart

Works..

mitchellkrogza commented 7 years ago

@BerndHelber awesome thanks. I will start pushing this new version out with the generator scripts. 👍

mitchellkrogza commented 7 years ago

I also ran the ruleset / access control layout structure through with some of the folks on #httpd on freenode and they all seemed to agree the new structure is correct and in accordance with the news 2.4 Specs.

mitchellkrogza commented 7 years ago

@BerndHelber the new 2.4 version is now being generated by Travis and is now up to date with all the latest bots and referrers. It also has it's own install and update scripts to pull from the correct section of the repo.

This is now at : https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/tree/master/Apache_2.4

Please test and let me know if you have any issues.

mitchellkrogza commented 7 years ago

@BerndHelber please pull latest version from https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/tree/master/Apache_2.4/custom.d updated today.

Some changes to the structure of the and blocks and commenting added in globalblacklist.conf.

This version now fully passes all Travis build tests so this means this is 100% good to go into a production environment on any Apache 2.4 server. I have Travis CI testing this on 2.4.27 and I have tested it on 2.4.18 too. Passing with flying colors now.

Thanks to you we now have a fully compliant version for Apache 2.4 without mod_access_compat. 👍 💯 🥇

BerndHelber commented 7 years ago

I can confirm everything works as described. :-) Great Work!

mitchellkrogza commented 7 years ago

Thanks @BerndHelber and thanks for the feedback very much appreciated.