search

mitchellkrogza / nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail for Repeat Offenders
Other
3.97k stars 477 forks source link

Remove archive.org_bot bad-user-agents.list #454

Closed MaximeMichaud closed 2 years ago

MaximeMichaud commented 2 years ago

https://archive.org/details%2Farchive.org_bot%2F https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/pull/183

mitchellkrogza commented 2 years ago

You can whitelist it yourself in https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/bots.d/blacklist-user-agents.conf (both white & blacklist) it will remain blocked unfortunately unless you whitelist it yoursefl.

GitHub
nginx-ultimate-bad-bot-blocker/blacklist-user-agents.conf at master · mitchellkrogza/nginx-ultimate-bad-bot-blocker
Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail f...
MaximeMichaud commented 2 years ago

So, I will Whitelist archive.org on hundred of host. Unfortunately.

mitchellkrogza commented 2 years ago

Never had a complaint yet in all these years and this blocker was built for my own needs and then made pubic. Most site owners do not want their site crawled by archive.org. It should take you all of 2 minutes with the command line to roll out a customized version of the whitelist file to thousands of sites. Sorry but it remains as is.

MaximeMichaud commented 2 years ago

I was not asking for any modification after the pull was closed :) Nevertheless, I am not the only one that may think that archive.org should not be blocked.