elixir-geolix / geolix

IP information lookup provider
Apache License 2.0
189 stars 18 forks source link

Is it possible to read the list of possible countries/cities? #22

Closed chrism2671 closed 5 years ago

chrism2671 commented 5 years ago

The use case is to make a dropdown to select a country from a list, to match with the Geolix discovered IP. It would be better to get this list from Geolix, rather than some separately maintained list.

mneudert commented 5 years ago

The MMDB2 database format usable is built to provide an "ip to city/country"-mapping and nothing else. So no, you currently cannot get a list of available countries from the database file directly.

Your best bet might be some library like chercheville that is able to provide an interface to the data from GeoNames. You could then use the geoname_id from a city or country lookup result to connect the databases. Depending on the data you might instead use the :iso_code from the country lookup result.

Or there is countries providing a different list you could connect using :iso_code or :geo (have not checked the data in regards to what field maps to what).


If you are interested in brute forcing the database to periodically extract a country list you could walk the full data part of your database. Shove every possible pointer (read as "0 to length of data") through MMDB2Decoder.value and check if the result is a country dataset. Then you could extract the countries and store the uniques.

A different approach would be to create a Geolix.Adapter implementation for the MaxMind CSV databases using for example an Ecto backend. Then you could use Ecto.Repo to fetch all unique countries from the database while still having all lookup features.

mneudert commented 5 years ago

For the sake of reference this is what "worked for me" to brute force all available countries:

{_meta, _tree, data} = MMDB2Decoder.parse_database(File.read!("/path/to/Country.mmdb"))
offsets = Range.new(0, byte_size(data) - 1)

offsets
|> Enum.map(fn offset ->
  try do
    MMDB2Decoder.Data.value(data, offset)
  rescue
    _ -> nil
  end
end)
|> Enum.filter(&is_map/1)
|> Enum.filter(&Map.has_key?(&1, :iso_code))

It uses the non-public module MMDB2Decoder.Data directly so perhaps not very future proof. But the extraction of all country records was done rather fast if you are willing to take that risk.