search

HurricaneLabs / machinae

Machinae Security Intelligence Collector
MIT License
504 stars 101 forks source link

Spamhaus 503 Errors #23

Open ryanohoro opened 8 years ago

ryanohoro commented 8 years ago

The Spamhaus Domain BL checks seem to be failing with a 503 error. I suspect this may be a result of Spamhaus's anti-automation checks.

[!] Error from Spamhaus Domain BL: 503 Server Error: Service Temporarily Unavailable for url: https://www.spamhaus.org/query/domain/www.google.com

The same URL in the browser works as expected, but fetching the HTML page is not the preferred method of checking the Spamhaus RBL. The appropriate way to query Spamhaus is via the DNSRBL.

Sample multi-DNSRBL resolver: http://pastebin.com/sCPeqtum

List of DNSRBLs: https://mxtoolbox.com/problem/blacklist/

mcm commented 8 years ago

Thanks @ryanohoro - I had noticed the Spamhaus 503's. I've been debating with myself the best way to add support for DNSRBL style stuff. I hate the way I did IP whois (using a python module for it is, in my mind, a cop-out, since its not a configurable service like the rest are), so I want to make sure whatever I do for Spamhaus is reusable for other sites too. I have some pretty big changes in mind for 2.0 as well, and part of me (selfishly) doesn't want to add a new site type until I've worked on all of that, especially because Spamhaus is (currently) the only service we have that would use it. That said, since Spamhaus is totally broken right now, some solution is probably in order sooner than that.

Any thoughts?

ryanohoro commented 8 years ago

The DNSRBL model is very mature and you can pick from well over 100 sources. It's fast and you don't need API keys. I'm sure there may already be some code that's more robust than mine, as there's even RFCs on how to structure DNSRBLs. Virtually every anti-spam and anti-malware solutions use them of course.

So, I'd say allow the user to configure or choose from a set of DNSRBLs in the conf, pick like 6 popular but diverse defaults. It really depends on the ultimate goals for the tool, what types of detections you're looking to find.

If you don't want to go to the trouble of collecting the response types, you can register a generic "block" on any response that's not NXDomain, which my code sample allows for.

mcm commented 8 years ago

@ryanohoro I'm going to work this weekend on adding a DNSRBL site type for these types of sites. I'll add a couple sources, and once the support is there I'd love if you submitted a few that you thought would be useful to include.

I'll let you know when I've got something available for you to look at.