search

elceef / dnstwist

Domain name permutation engine for detecting homograph phishing attacks, typo squatting, and brand impersonation
https://dnstwist.it
Apache License 2.0
4.82k stars 765 forks source link

Reorder adding domains to job queue for speedup #33

Closed andrewjbennett closed 8 years ago

andrewjbennett commented 8 years ago

Creating the queue and adding the domains to it before creating the jobs performs significantly faster for me than doing it the other way around. I'm not enough of an expert with Python to be sure why this is, but it's possible there's some overhead in adding items to the queue once every thread has a reference to it, which isn't there when you add everything to the queue before assigning jobs to use it.

elceef commented 8 years ago

I've made some quick benchmarks and can't see any difference. Could you please do some on your side? What's your configuration?

andrewjbennett commented 8 years ago

I've been running on lists of 500000 domains (after fuzzing and removing duplicates), on both OSX on my laptop and Ubuntu 14.04 on an AWS instance.

I can send you benchmarks if that would still be helpful.

On Tue, 2 Feb 2016 19:21 Marcin Ulikowski notifications@github.com wrote:

I've made some quick benchmarks and can't see any difference. Could you please do some on your side? What's your configuration?

— Reply to this email directly or view it on GitHub https://github.com/elceef/dnstwist/pull/33#issuecomment-178439947.

elceef commented 8 years ago

No need. Thank you!