Parallelize DNS lookups

mlsecproject / combine

Tool to gather Threat Intelligence indicators from publicly available sources

https://www.mlsecproject.org/

GNU General Public License v3.0

654 stars 171 forks source link

Parallelize DNS lookups #38

Open krmaxwell opened 10 years ago

krmaxwell commented 10 years ago

Not everybody has DNSDB type availability. Fail gracefully if they don't, preferably with some other source.

krmaxwell commented 10 years ago

Also: speedups by making requests in parallel or using sessions / pipelining?

krmaxwell commented 9 years ago

The original part of this is complete, actually. But we do need to do the speedup.

juju4 commented 9 years ago

2 comments:

free alternative: https://github.com/mlsecproject/combine/issues/144 and existing python module based on rfc draft (common with farsight) https://github.com/CIRCL/PyPDNS
implement requests-cache in grequests (basic, need 2 lines of extra code: https://pypi.python.org/pypi/requests-cache)

juju4 commented 9 years ago

On DNS resolution, I think 1- it should be optional (be it for performance vs passivedns info, be it for opsec or other reason) 2- can switch to adns in async mode but should be bulk call not one to one http://www.catonmat.net/blog/asynchronous-dns-resolution/