search

0xd34db33f / gfyp

Unification of dnstwist + SQLite + Email reporting. Set it as a cron job that runs every hour, give it a list of domains and email addresses for reporting, then watch it go find stuff.
66 stars 12 forks source link

bug: number of domains found is not consistent #10

Open kristovatlas opened 7 years ago

kristovatlas commented 7 years ago

Something I noticed when suffering from #9 is that the number of domain variants within a short period of time is inconsistent. This is probably some DNS quirk but might merit more research.

Here's output from a single invocation:

2017-01-16 19:21:16,065:INFO:Now checking alert@example.com - example.com
2017-01-16 19:21:17,431:INFO:DNSTwist found 71 variant domains from example.com.
2017-01-16 19:21:17,433:INFO:Found 0 new domain variants from example.com
2017-01-16 19:21:17,433:INFO:Now checking alert@example.com - example.com
2017-01-16 19:23:30,415:INFO:DNSTwist found 68 variant domains from example.com.
2017-01-16 19:23:30,416:INFO:Found 0 new domain variants from example.com
2017-01-16 19:23:30,416:INFO:Now checking alert@example.com - example.com
2017-01-16 19:23:30,888:INFO:DNSTwist found 70 variant domains from example.com.
2017-01-16 19:23:30,890:INFO:Found 0 new domain variants from example.com
2017-01-16 19:23:30,890:INFO:Now checking alert@example.com - example.com
2017-01-16 19:23:46,608:INFO:DNSTwist found 62 variant domains from example.com.
2017-01-16 19:23:46,610:INFO:Found 0 new domain variants from example.com
2017-01-16 19:23:46,610:INFO:Now checking alert@example.com - example.com
2017-01-16 19:25:22,057:INFO:DNSTwist found 73 variant domains from example.com.
2017-01-16 19:25:22,059:INFO:Found 0 new domain variants from example.com
2017-01-16 19:25:22,059:INFO:Now checking alert@example.com - example.com
2017-01-16 19:25:22,570:INFO:DNSTwist found 73 variant domains from example.com.
2017-01-16 19:25:22,571:INFO:Found 0 new domain variants from example.com
2017-01-16 19:25:22,572:INFO:Now checking alert@example.com - example.com
2017-01-16 19:25:32,913:INFO:DNSTwist found 72 variant domains from example.com.
2017-01-16 19:25:32,915:INFO:Found 0 new domain variants from example.com
2017-01-16 19:25:32,915:INFO:Now checking alert@example.com - example.com
kristovatlas commented 7 years ago

Note that for anyone who can tolerate waiting a few extra cron invocations, this doesn't matter much since it's stable enough over time to catch domains that get "missed" in a single invocation. Time-until-seen can be decreased by running cron job more often.

0xd34db33f commented 7 years ago

I've seen this before as well and just chalked it up to DNS rate limits (we're pretty much loading up a bunch of domains and firing it off which I've actually saturated my test DSL line before). If we introduce any internal rate limits I'm fine with that, we'll have to start modifying the dnstwist code though to handle it.