Deny Semalt botnet spam

omega8cc / boa

Barracuda Octopus Aegir 5.4.0

https://omega8.cc/compare

394 stars 75 forks source link

Deny Semalt botnet spam #490

Closed pricejn2 closed 9 years ago

pricejn2 commented 9 years ago

Can we add Semalt as a denied crawler in /var/aegir/config/server_master/nginx.conf?

See http://www.incapsula.com/blog/semalt-botnet-spam.html

omega8cc commented 9 years ago

Does this really use the same UA identity?

omega8cc commented 9 years ago

The answer is no, we can't stop it like this, because they use victims browsers, so requests look like this:

"190.185.x.x, 127.0.0.1" omega8.cc [12/Nov/2014:20:04:17 +0100] "GET / HTTP/1.0" 200 3410 511 3984 "http://semalt.semalt.com/crawler.php?u=http://aegir.us" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36" 0.159 "3.87"

omega8cc commented 9 years ago

It can be blocked only by checking HTTP_REFERER, and they have growing collection of domains.

omega8cc commented 9 years ago

Added in https://github.com/omega8cc/provision/commit/34900d32be6b4b3aa492bf11c1c893307b9a6114

pricejn2 commented 9 years ago

Ah yes, of course they wouldn't use a consistent UA.

What you've added looks perfect, though. Thanks!