[REMOVE] "(?:\b)facebookexternalhit(?:\b)" good_bot

mitchellkrogza / apache-ultimate-bad-bot-blocker

Apache Block Bad Bots, (Referer) Spam Referrer Blocker, Vulnerability Scanners, Malware, Adware, Ransomware, Malicious Sites, Wordpress Theme Detectors and Fail2Ban Jail for Repeat Offenders

Other

799 stars 173 forks source link

[REMOVE] "(?:\b)facebookexternalhit(?:\b)" good_bot #208

Open doxycomp opened 1 month ago

doxycomp commented 1 month ago

Removal Request?

Please List the User-Agent string or Referrer to be added/removed

/globalblacklist.conf:BrowserMatchNoCase "(?:\b)facebookexternalhit(?:\b)" good_bot

Facebook is not a good bot and ignoring robots.txt and spamming the server with requests.

For Additions: Please include a log sample 3-5 lines is adequate

It defaults to good_bot in globalblacklist.conf and seemingly cannot be set as bad_bot in blacklist-user-agents.conf

Thank you!

doxycomp commented 1 month ago

Just in case, sombody else is struggling with this, for the moment we added some lines to the top of our .htaccess, to block the facebook/meta crawlers:

RewriteCond %{HTTP_USER_AGENT} ^facebookexternalhit. [OR] RewriteCond %{HTTP_USER_AGENT} ^meta-externalagent. RewriteRule .* - [F,L]

Kind regards :)