search

mitchellkrogza / nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail for Repeat Offenders
Other
3.82k stars 472 forks source link

[User-Agent] 'Domains Project' Crawler - Bad Bot #378

Closed Jas0n99 closed 4 years ago

Jas0n99 commented 4 years ago

Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)


Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)

Is this for Addition / Removal?

Did the User-Agent request robots.txt first?

Post Log Excerpt to show User-Agent behavior (10-20 lines is enough)

82.193.104.168 - - [25/Apr/2020:06:28:50 -0500] "GET / HTTP/2.0" 200 11876 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:51 -0500] "GET /aggregate_summary.php?s= HTTP/2.0" 200 7981 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:51 -0500] "GET /?nav=FAQ HTTP/2.0" 200 6499 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:51 -0500] "GET /?nav=XML HTTP/2.0" 200 6001 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:51 -0500] "GET /?nav=IMAGES HTTP/2.0" 200 5980 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:51 -0500] "GET /?nav=DBINFO HTTP/2.0" 200 4524 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /search.php HTTP/2.0" 200 4066 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /team_overtake.php?s=&t=167809 HTTP/2.0" 200 12035 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /user_list.php?s=&t=167809 HTTP/2.0" 200 11330 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /individual_list.php?s= HTTP/2.0" 200 11778 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /team_list.php?s= HTTP/2.0" 200 15689 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /team_summary.php?s=&t=167809 HTTP/2.0" 200 7776 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /team_summary.php?s=&t=239945 HTTP/2.0" 200 7663 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /user_summary.php?s=&u=471536 HTTP/2.0" 429 134 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /user_summary.php?s=&u=658640 HTTP/2.0" 429 134 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /user_summary.php?s=&u=511665 HTTP/2.0" 429 134 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /user_summary.php?s=&u=625303 HTTP/2.0" 429 134 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /user_summary.php?s=&u=692199 HTTP/2.0" 429 134 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /user_summary.php?s=&u=616207 HTTP/2.0" 429 134 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /team_summary.php?s=&t=44627 HTTP/2.0" 200 6737 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /team_summary.php?s=&t=153191 HTTP/2.0" 200 6527 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /user_summary.php?s=&u=836546 HTTP/2.0" 429 134 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /user_summary.php?s=&u=750215 HTTP/2.0" 429 134 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /user_summary.php?s=&u=611029 HTTP/2.0" 429 134 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"
82.193.104.168 - - [25/Apr/2020:06:28:52 -0500] "GET /user_summary.php?s=&u=1000797 HTTP/2.0" 429 134 "-" "Mozilla/5.0 (compatible; Domains Project/1.0.6; +http://domainsproject.org)"

Additional information

Never requested robots.txt, and query rate ramped up to ~40 pages/second. Thankfully I have rate limiting so most requests got a 429 back... Sad thing is this crawler is another GitHub project (and crawler is using github IPs), I posted an issue on his project page and he doesn't seem to think there is any problem with what he is doing... 🙄

mitchellkrogza commented 4 years ago

Thanks @Jas0n99 they have been added, I checked out their site and it appears to be a totally useless project, just a list of domains which you have already proven wrong as they are crawling more than just domain names.