GeoIp2 replace GeoIP

[mmorel-35]

GeoIP .dat file will not be used anymore, the application shall handle .csv or .mmdb files

We will be discontinuing updates to the GeoLite Legacy databases as of April 1, 2018. You will still be able to download the April 2018 release until January 2, 2019. GeoLite Legacy users will need to update their integrations in order to switch to the free GeoLite2 or commercial GeoIP databases by April 2018.

For more information, please visit our Support Center.

In addition, in 2019, latitude and longitude coordinates in the GeoLite2 databases will be removed.* Latitude and longitude coordinates will continue to be provided in GeoIP2 databases. Please check back for updates.

[kinmenRP]

any news about that?

[neilgierman]

I have started working on one solution. Creating a GeoIP2 plugin that is based off the GeoIP plugin. Feel free to check it out. I have tested it and it works using the GeoLite2-Country database for looking up both IP4v and IPv6 addresses. Comments are welcome.

https://github.com/neilgierman/awstats/tree/GeoIP2

[kinmenRP]

Good. Unfortunately I can't test it, I still use an older version installed from packages manager on a production site, but I hope to see the new working plugin soon. Tnx for your efforts

[neilgierman]

Yes. I have the 1.5 version of GeoIP installed from the production repositories in CentoOS7. I installed the GenIP2 version of geoipupdate from https://github.com/maxmind/geoipupdate which installed to /usr/local/bin, and the mmdb files are downloaded to /usr/local/share/GeoIP so it doesn't conflict with the 1.x versions installed from production repositories.

The one challenge I saw is there is not a GeoIP2 version of the Perl module any of the CentOS supported repositories (EPEL, Atomic, etc), so I had to install the required module directly from CPAN.

[kinmenRP]

I mean, I have awstats 7.6. I've cpan install GeoIP2 but, where's geoip2.pm? It's not in /usr/share/awstats/plugins/

[neilgierman]

To manually install, you would need to copy geoip2.pm (and geoip2_city.pm if you want) into /usr/share/awstats/plugins and also you would need to patch /usr/share/awstats/wwwroot/cgi-bin/awstats.pl with the other changes I made to call the new plugin. Finally you would need to modify your conf file to load the geoip2 plugin(s) pointing to their respective mmdb files instead of the existing geoip plugins (examples are in awstats.model.conf).

Please keep in mind that I have done my testing only on a complete source package based off 7.7 (HEAD of the develop branch). I don't know if there would be other issues if you manually patched a 7.6 installation with my changes.

[kinmenRP]

Yes, but the problem is that after doing cpan install GeoIP2 I can't find geoip2.pm anywhere... Do you mean I have to copy it from /usr/local/share/perl/5.22.1/GeoIP2.pm ?

[neilgierman]

I think there may be some confusion. The CPAN install is only to install the ability for Perl programs to query the new GeoIP2 mmdb databases. The geoip2.pm that I created in my code is an awstats plugin to call the CPAN module with specific instructions on how to get information out of the mmdb files. The flow is awstats.pl (my modified version) -> geoip2.pm (my file) -> CPAN Module (in the default Perl INC path) -> mmdb database.

[kinmenRP]

You're right, I'm in confusion... Sorry some years have passed since the last time I've put my hands on awstats :-P I've copied your awstats.pl into my cgi folder, modified my awstats.conf as described. Already have the new db in the GeoIP folder, now let's see what will happen on the next update... Thanks for your patience

[kinmenRP]

It "works", but the generated page stops at the host info; I see the countries' graph, but no list of countries & cities (excepted one host that shows: The IP address you provided (xxxxxxx) is not a valid IPv4 or IPv6 address at /usr/local/share/perl/5.22.1/MaxMind/DB/Reader/Role/Reader.pm line 39). Also all the sections like visited urls, the errors, browsers, etc. have disappeared: I see that the above pages are generated, but not attached to the "main" page. Any clue about what happened?

[neilgierman]

Yes. I see the issue. It seems (I am still learning awstats) that in the ShowHost section, the resolved DNS name is sent instead of the IP address. I have updated that section of geoip2.pm to do the resolution from name back to IP since the GeoIP2 API only supports IPv4 and IPv6 addresses, not hostnames. I'm sure there is a more efficient way to do this from dnscache but it should at least get you unblocked.

[kinmenRP]

I don't see that error anymore now. However I've just found out that the geoip2 plugin for cities doesn't work, doesn't generate any html file (error 404, file is really missing inside the dir). By the way I'm still missing all the bottom section of the main file (just as reference, all the part under the cake graph of the Hosts)

[neilgierman]

Yes. Probably best to not load the geoip2_city plugin for now. I checked it in only to save my work but not read for testing just yet. Trying to clean up the country plugin first before I get back to the city plugin.

[kinmenRP]

Ok, I've disabled it

[sherpya]

in the meanwhile, why non making a script that converts geolite2 db in the legacy format?

[sherpya]

ok I wrote a tool to convert geolite2 to old format https://github.com/sherpya/geolite2legacy

[kinmenRP]

In these days I don't have the time to try it, but I'll asap. Thank you!

[CEric-fr]

Try you converter sherpya, go a error (see issue on your github) on city base. Wait for the city plugin to work ...

[ffrediani]

Hi there. Do I understand correct but with the current status I should keep using the AWStats GeoIP (or GeoIP6) Plugin and convert the Geolite2 database to the old format using @sherpya tool ?

I guess that will be mainly valid after 2nd of January of 2019 when Maximind will remove it from their website.

[CEric-fr]

The @sherpya converter (V2->V1) work for country and city but not for region, and this is what we need (region). Stop testing converter because no time to spend for it... Wait for city and region plugin (with geolite2) to work...

[kinmenRP]

@ffrediani You're right. I finally have got some time to test the script, on two different servers, with the free GeoLite2 Country & City. No error reported, seems working fine.

[sherpya]

not quite yet ready, I'm working on a tool that maps maxmind ids to old fips-10-4 country and region codes

[sherpya]

I've added a partial location map please follow the issue directly and test if it you can https://github.com/sherpya/geolite2legacy/issues/1

[CEric-fr]

Create a branche to resolve blocking error from Maxmind geoip package, all refused IP or not in database are 'unknown'.

Just 'geoip2.pm' modified : https://github.com/CEric-fr/awstats/blob/patch-1/wwwroot/cgi-bin/plugins/geoip2.pm

[mrlerch]

Thank you for starting to adapt to the new geoip2 format. I see that you now have geoip2.pm and geoip2_city.pm. How about the new geoip2_asn.pm? Any news on that? Or is this project dead?

[k2patel]

@neilgierman is there any plan for the pull request or upstream merge for the issue ? I do see very good solution except for not working asn, it does works for my testing on CentOS 8. I would love to see if it get merged and become part of the upstream. Thanks

[neilgierman]

From what I can tell, all outstanding issues were resolved with https://github.com/eldy/awstats/pull/131 by @manuelm . If there are additional issues with HEAD, please open a new Issue.

[ffrediani]

So what is missing in order for GeoIp2 to work natively with awstats without having to be converted back to v1 ? The ASN list is also something pretty usable in many scenarios.

[neilgierman]

To use the GeoIP2 plugin you will need to install the GeoIP2 module from CPAN (unless your distro already has a package for it) and update your awstats.*.conf to load the appropriate plugin and parameters. I don't use the ASN list in my setup so I didn't include it in my implementation.

[ffrediani]

So what does it take to include the ASN list as I believe it is something used by many setups ?

[neilgierman]

Someone will need to contribute that code to awstats. As this is an open source project anyone can fork the repository, create new functionality, and create a pull request to get it in mainline.

[ffrediani]

Pretty obvious I guess. My question, more objective, was in terms of coding and functions to make that work in line with already is implemented and working for GeoIp2.

[neilgierman]

What I did with the base and city module was look at how GeoIPv1 was done, duplicate that functionality and only where it was reading from the GeoIPv1 database, make changes to read from the GeoIPv2 database.

[ffrediani]

Anyone would be willing to get ASN support for GeoIP2 as well ?

[ghnp5]

Hey - if someone could provide me some help in #150, that would be very great! Thanks very much :)

[neilgierman]

@ghnp5 Just replied in that thread.

[ffrediani]

Why has this issue been closed without any actions ?

[mmorel-35]

@eldy, this issue is 3 years old and no fix have been apported. What's the point of keeping it opened? I reopen it so you can close it yourself with or without fix, it's up to you

[mmorel-35]

@ffrediani, I moved to Matomo, which is an alternative to AWStats that supports natively GeoIp2. I couldn't wait so long for a fix

[ffrediani]

Hi @mmorel-35 I understand what you mean but I think that while it is an issue it needs to be kept opened until can be loked at, otherwise it is just sweep under the carpet what will continue to be an issue.

Thanks for the tip about Matomo.

[ghnp5]

I agree with @ffrediani - I never understood what's the point of closing old "issues", when they are still relevant and valid on the day.