search

elceef / dnstwist

Domain name permutation engine for detecting homograph phishing attacks, typo squatting, and brand impersonation
https://dnstwist.it
Apache License 2.0
4.81k stars 764 forks source link

Not all second level domains are detected. #98

Closed hackthebox1 closed 3 years ago

hackthebox1 commented 4 years ago

While running dnstwist against alexa rank 155 domain "hatena.ne.jp" only the "ne" portion of the domain was modified in the output. Behavior against "bbc.co.uk" was as expected, with the "bbc" portion being modified. Similar to "co.uk", "ne.jp" is not available for use by users as explained here https://www.ne.jp/ . "ne" is a second level domain for ".jp" and as such the "hatena" portion of the query should have been modified. The list of all common second level domains can be found on wikipedia https://en.wikipedia.org/wiki/Second-level_domain . I am not certain if this problem happens across other second level domains, but it would improve the tool if all common second level domains were detected.

elceef commented 4 years ago

You can correct this by installing tld Python package. If not present, the tool uses a simplified algorithm which works pretty well in general, but can miss some uncommon second-level domains.