UnderstandLingBV / Tuktu

Big Data Science Swiss Army Knife - http://www.tuktu.io --
http://www.understandling.com
Other
60 stars 16 forks source link

Make a processor which uses an open source DB of IP to Country #44

Closed NTPape closed 8 years ago

NTPape commented 8 years ago

Currently the only way to get the Country of a given IP in Tuktu is to use external dependencies like FreeGeoIP.net, which is not 100% reliable and slow.

Instead make a processor which uses a local open source lookup table to do so.

dmassart commented 8 years ago

You should not use the demo instance of freegeoip in production which is indeed limited. Instead, you should download the freegeoip app, deploy it on a local server, and point your processor to this local instance. It dramatically improves both speed and reliability.

NTPape commented 8 years ago

Yes, good point.

But this is more a matter of having external dependencies at all, especially if they have to be run separately as a server. Having another server running just for IP to Country DB seems a little over the top. There are plenty of open source CSV files out there which do just that, it should not be too hard to read it, index it, and use it to lookup the correct country. I'm pretty sure that the FreeGeoIP app does the same internally anyway.

dmassart commented 8 years ago

In my opinion, the main problem with this approach comes from the fact that geo-localisation of IPs changes all the time and it will be necessary to very regularly update your CSV (which creates de facto a dependency). In my opinion, one of the main advantages of the FreeGeoIP approach is that it takes care of this for you. Each client stores its data in a local database that is transparently kept up-to-date by synchronizing data from a central service. I have a tuktu cronjob that has been happily running and resolving IP locations for more than 6 months now without having had to touch to it in any way ;-)

ErikTromp commented 8 years ago

I agree with both: either get a CSV dump and use the processors already in Tuktu, or deploy a local instance of a service like Free Geo IP. Closing.