search

mitchellkrogza / apache-ultimate-bad-bot-blocker

Apache Block Bad Bots, (Referer) Spam Referrer Blocker, Vulnerability Scanners, Malware, Adware, Ransomware, Malicious Sites, Wordpress Theme Detectors and Fail2Ban Jail for Repeat Offenders
Other
826 stars 181 forks source link

[BUG] Cannot block "good" User-Agent in blacklist-user-agents.conf #204

Open MyLifeOrganized opened 5 months ago

MyLifeOrganized commented 5 months ago

I want to block an User-Agent which is listed as "good" in globalblacklist.conf

However when I add it to blacklist-user-agents.conf it is not blocked. Steps to reproduce:

  1. add this line to blacklist-user-agents.conf: 
    BrowserMatchNoCase "\bApplebot\b" bad_bot

(Applebot is a "good bot" in globalblacklist.conf)

  1. Restart apache

  2. Try:

    curl -A "Applebot" https://yoursite.net

Expected:  403 Forbidden

Actually: The HTML page is returned normally

I am using version Apache_2.4

I also tried this syntax and it did not work as well BrowserMatchNoCase "(?:\b)Applebot(?:\b)" bad_bot

If I block other User-Agents not white-listed in globalblacklist.conf it works as expected and blocked

Your documentation in blacklist-user-agents.conf says:

# Please note this include file loads first before any of the already whitelisted User-Agents
# in the bad bot blocker. By loading first in line it over-rides anything below it so for instance
# if you want to block Baidu, Google or Bing for any reason you add them to this file which loads
# first and takes precedence over anything below it. This now allows even finer control over the 
# bad bot blocker. Enjoy !!!

I see that actually blacklist-user-agents.conf is included after, not before white listing in globalblacklist.conf I've also tried to change globalblacklist.conf  and include blacklist-user-agents.conf first and it still does not work.

How do I block an User-Agent white-listed in globalblacklist.conf?

Server :

kasutajaimin commented 2 months ago

Can confirm same behaviour on Apache 2.4.6 @ CentOS 7.9

FlowLaGit commented 10 hours ago

It seems that the variable "good_bot," which was set using "BrowserMatchNoCase," cannot simply be overwritten again with "BrowserMatchNoCase."

I was able to solve this by removing the "good_bot" variable with SetEnvIfNoCase User-Agent "\b$BOTNAME\b" !good_bot and then resetting it with BrowserMatchNoCase "\b$BOTNAME\b" bad_bot.

Here is an example from my blacklist-user-agents.conf with the YandexBot:

    # Removing the "good_bot" flag and setting the "bad_bot" flag.
    # -----------------------------
    SetEnvIfNoCase User-Agent "\bYandexBot\b" !good_bot
    BrowserMatchNoCase "\bYandexBot\b" bad_bot
    # -----------------------------