search

mitchellkrogza / nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail for Repeat Offenders
Other
3.97k stars 477 forks source link

[User-Agent] Mozilla/5.0 (compatible; Adsbot/3.1) #397

Open saschadd opened 4 years ago

saschadd commented 4 years ago

Paste the full User-Agent String here

"Mozilla/5.0 (compatible; Adsbot/3.1)"

Is this for Addition / Removal?

Did the User-Agent request robots.txt first?

Not sure but ignores my robots.txt entry for this specific user agent string.

Post Log Excerpt to show User-Agent behavior (10-20 lines is enough)


Post log excerpt here (paste in between the ```     ``` markers)

Additional information

Seems to scan/scrape our whole website page by page for unknown reason.

saveman71 commented 4 years ago

Same here, vote to add

zjagust commented 3 years ago

Also vote to add, according to Herr Bischoff's Bot Database, it does not respect robots.txt.