search

elceef / dnstwist

Domain name permutation engine for detecting homograph phishing attacks, typo squatting, and brand impersonation
https://dnstwist.it
Apache License 2.0
4.81k stars 764 forks source link

Replace GeoIP Legacy by GeoIP2 #120

Closed paravoid closed 2 years ago

paravoid commented 3 years ago

The GeoIP module, nowadays called "GeoIP Legacy", alongside the old-style databases (.dat files) are deprecated, and have been for some time. The free-as-in-beer databases in that format (GeoLite Legacy) were discontinued in January 2019, and the paid GeoIP Legacy versions are being retired in May 2022.

The replacement stack exists since 2014, and is by the same author (MaxMind), and fairly mature. These comprise the MMDB file format (an openly licensed, well-documented format), alongside the MaxMindDB libraries, in C, Python etc. In Python, the MaxMindDB library is an abstraction for reading the databases on disk (like the old GeoIP module was), while the GeoIP2 module is one level of abstraction further, allowing one to use either the on-disk databases or their web services instead.

I was wondering if dnstwist could be adapted to use GeoIP2. Thank you for your efforts!

mda1125 commented 3 years ago

Wondering that myself! Tried just changing the script to import geoip2 but that didn't work. Seems like it's possible but I haven't figured it out yet either

elceef commented 3 years ago

Preliminary support for MaxMind GeoIP2 Lite has been just added. The tool should switch to GeoIP2 if geoip2 module is installed and GeoLite2-Country.mmdb database file can be found in the main directory. Otherwise, it will fallback to the GeoIP Legacy. Obviously, due to MaxMind GeoIP2 license, the database file can't be shipped with the tool and has to be provided manually by the user.