search

mitchellkrogza / nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail for Repeat Offenders
Other
3.81k stars 472 forks source link

[User-Agent] AwarioBot #560

Closed robwent closed 3 months ago

robwent commented 3 months ago

Paste the full User-Agent String here


Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)

Is this for Addition / Removal?

Did the User-Agent request robots.txt first?

Post Log Excerpt to show User-Agent behavior (10-20 lines is enough)


46.4.108.51 - - [28/Mar/2024:06:11:19 +0000] "GET /yachts-for-sale-by-model/lagoon-450-s/ HTTP/2.0" 200 31238 "-" "Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)"
46.4.108.51 - - [28/Mar/2024:06:11:23 +0000] "GET /yachts-for-sale/pilot-lounge-name-reserved-2/ HTTP/2.0" 200 38904 "-" "Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)"
46.4.108.51 - - [28/Mar/2024:06:11:31 +0000] "GET /yachts-for-sale/riva-ariston/ HTTP/2.0" 200 30524 "-" "Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)"
46.4.108.51 - - [28/Mar/2024:06:11:35 +0000] "GET /yachts-for-sale-by-model/lagoon-450/ HTTP/2.0" 200 31121 "-" "Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)"
46.4.108.51 - - [28/Mar/2024:06:11:38 +0000] "GET /yachts-for-sale/gatsby/ HTTP/2.0" 200 32968 "-" "Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)"

Additional information

A brand monitoring company scanning billions of pages a day: https://awario.com/

mitchellkrogza commented 3 months ago

Hi @rogerdz companies that do brand surveillance do thousands of scans on people's sites and are generally considered a nuisance and the intention on this blocker is to block nuisances.

Companies that also use this data to sell services (see their pricing pages) are again using people's bandwidth and resources to earn money selling a service. Never had any complaints of these being blocked so whitelisting them for your own purposes is as simple as adding their User Agent names to your whitelist

https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/bots.d/blacklist-user-agents.conf

GitHub
nginx-ultimate-bad-bot-blocker/bots.d/blacklist-user-agents.conf at master · mitchellkrogza/nginx-ultimate-bad-bot-blocker
Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail f...
mitchellkrogza commented 3 months ago

Brand monitoring bots are no different to SEO company bots all of which are banned and will continue to be banned as they change UA, IP and domain names.

robwent commented 3 months ago

Were the comments meant for #559?

I see 2 related useragents in the global blacklist, but maybe they recently changed it?

https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/master/conf.d/globalblacklist.conf#L182-L183

GitHub
nginx-ultimate-bad-bot-blocker/conf.d/globalblacklist.conf at master · mitchellkrogza/nginx-ultimate-bad-bot-blocker
Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail f...