search

mitchellkrogza / nginx-ultimate-bad-bot-blocker

Nginx Block Bad Bots, Spam Referrer Blocker, Vulnerability Scanners, User-Agents, Malware, Adware, Ransomware, Malicious Sites, with anti-DDOS, Wordpress Theme Detector Blocking and Fail2Ban Jail for Repeat Offenders
Other
3.97k stars 477 forks source link

[User-Agent] (SeekPort Crawler) #444

Closed Loforo closed 2 years ago

Loforo commented 3 years ago

Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/)

Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/)

Is this for Addition / Removal?

Did the User-Agent request robots.txt first?

Post Log Excerpt to show User-Agent behavior (10-20 lines is enough)

95.217.229.84 - - [28/Sep/2021:10:09:17 +0200] "GET /sc/acsT3M7h.png HTTP/1.1" 200 390918 "-" "'Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/'"
95.217.229.84 - - [28/Sep/2021:10:09:17 +0200] "GET /sc/2vK15d2N.png HTTP/1.1" 200 213663 "-" "'Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/'"
95.217.229.84 - - [28/Sep/2021:10:09:17 +0200] "GET /sc/C0Ez02ac.png HTTP/1.1" 200 606895 "-" "'Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/'"
95.217.229.84 - - [28/Sep/2021:10:09:17 +0200] "GET /sc/atMhliOG.png HTTP/1.1" 200 253559 "-" "'Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/'"
95.217.229.84 - - [28/Sep/2021:10:09:17 +0200] "GET /sc/zqSm9ixV.png HTTP/1.1" 200 307611 "-" "'Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/'"
95.217.229.88 - - [28/Sep/2021:10:24:02 +0200] "GET /sc/FJ93GPjR.jpg HTTP/1.1" 200 86146 "-" "'Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/'"
95.217.229.88 - - [28/Sep/2021:10:24:02 +0200] "GET /sc/K5hRa0IK.jpg HTTP/1.1" 200 77639 "-" "'Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/'"
95.217.229.88 - - [28/Sep/2021:10:24:02 +0200] "GET /sc/kPGTTgqX.jpg HTTP/1.1" 200 77849 "-" "'Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/'"
95.217.229.88 - - [28/Sep/2021:10:24:02 +0200] "GET /sc/4Iqo4M4R.jpg HTTP/1.1" 200 75634 "-" "'Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/'"
95.217.229.88 - - [28/Sep/2021:10:24:02 +0200] "GET /sc/5ulKSfWm.jpg HTTP/1.1" 200 87079 "-" "'Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/'"
95.217.229.88 - - [28/Sep/2021:10:24:02 +0200] "GET /sc/XEQ4ci7m.png HTTP/1.1" 200 479294 "-" "'Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/'"
95.217.229.88 - - [28/Sep/2021:10:24:02 +0200] "GET /sc/9r376mWN.jpg HTTP/1.1" 200 68278 "-" "'Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/'"
95.217.229.88 - - [28/Sep/2021:10:24:02 +0200] "GET /sc/03RPUzBJ.jpg HTTP/1.1" 200 75183 "-" "'Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/'"

Additional information

Seems the original SeekPort search engine went bankrupt and an SEO company acquired the domain.

This bot is crawling at a ridiculously high rate for hours on end eventually causing the server to become overloaded for some time.

Searching for this user-agent on Google reveals lots of others who have add this user agent to their robots.txt file.

After adding to the user agent blacklist and returning 444's, it is still crawling. In the past 10 hours alone it has crawled over 28000 files:

/var/log/nginx # grep -c seekport access-ts.log
28319