search

matomo-org / matomo

Empowering People Ethically with the leading open source alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites & apps and visualise this data and extract insights. Privacy is built-in. Liberating Web Analytics. Star us on Github? +1. And we love Pull Requests!
https://matomo.org/
GNU General Public License v3.0
19.87k stars 2.65k forks source link

Plugin using GeoIP to accurately detect visitor countries, cities, and continent #5465

Closed mattab closed 10 years ago

mattab commented 16 years ago

GeoIp Plugin

Requirements

You need at least Piwik 0.2.33 to run the GeoIp plugin.

How to install?

How to apply the GeoIp to all your previous visits to fix your previous reports

You can apply GeoIP precise localization on your past data:

Authors

Changelog

Feedback

Please leave a comment if you have any feedback, suggestion, or bug report. Keywords: third-party-plugin

robocoder commented 14 years ago

Estaban: download and install GeoIP, and apply the patch from Ian in comment:142

anonymous-matomo-user commented 14 years ago

Replying to vipsoft:

Estaban: download and install GeoIP, and apply the patch from Ian in comment:142

Replying to ian:

Turned out to be a very simple patch. I think this is low risk - it leaves location_country alone unless GeoIP knows what the country is. Ran it against the historical visits on my end and the Core and GeoIP "Visitor Countries" widgets are now much closer. No "unknowns" showed up in the Core widget, but India, which had been completely absent, is there now. (Pretty much everyone in India has their browser language set to English/US, apparently..)

I would appreciate feedback/review, this is my first submitted Piwik patch.

Excellent! problem solved. Thanks

But not as Piwik demo is not on the GeoIP plugin (no I'm not mistaken) yet recorded Ecuador South America

Why?

Esteban

robocoder commented 14 years ago

Country-to-continent assignment is via a lookup table. In Piwik 1.0, Central America is lumped in with South America.

For Piwik 1.1, we're using the Maxmind GeoIP mapping with the exception of Central America (which they lump in with North America). We've added Antarctica and Central America as separate continental regions.

lehrer commented 14 years ago

Replying to vipsoft:

Estaban: download and install GeoIP, and apply the patch from Ian in comment:142 Hi,

How do I apply the patch? I'm not used to work with diff files.

anonymous-matomo-user commented 14 years ago

Replying to poudro:

By applying the patch 'patch_GeoIP_0.7_google_map_link' I just uploaded to the original tree it adds links to google maps with the location of cities in the visitor countries dropdown...

hi! google-map-links are working, unfortunately piwik seems to accumulate the longitude and altitude numbers, so the links of cities with more than one visitors are pointing to nirvana... do you have an idea what i have to change in the code to correct this? thx!

anonymous-matomo-user commented 14 years ago

as you can see, loc data of Leonding is correct, but data of Vienna isn't. Has anybody an idea why? piwik is accumulating the longitude and altitude by archieving. but o don't know what i have to change...

anonymous-matomo-user commented 14 years ago

Hi,

When the Geoip is installed as a PECL extension on your server the plugin doesn't work. The included geoip library redefines functions that are already defined by the PECL extension.

I made a patch which makes the plugin use PECL extension when available instead of including the geoip library.

@see UsePeclExtension.patch

anonymous-matomo-user commented 13 years ago

Followed the instruction and installed plugin GeoIP. Also downloaded the lite .dat file.

Still all visitors are marked to come from the US.

Visitors -> Locations & Providers show no data for this report for both Country and Continent.

How do I check whether GeoIP plugin is configured correctly? It is enabled.

Much thanks!

anonymous-matomo-user commented 13 years ago

BTW, the time log of visitors visiting the site is correct under Vistors -> Visitor Log.

Thanks again!

anonymous-matomo-user commented 13 years ago

4 hours later the country automagically becomes 7 = unknown, 2 = asia, 1 = US. As I'm in Hong Kong, at least it now shows something other than US.

Does anyone know how long it takes for data to become available for reporting?

anonymous-matomo-user commented 13 years ago

Anyone please?

mattab commented 13 years ago

Note: before including GeoIP in trunk, GeoIP should be modified so that it records the new country and continent in place of the default one (columns location_*) , rather than in new columns locationgeoip in the log_visit table.

anonymous-matomo-user commented 13 years ago

Thanks Matt for the pointer.

How should I go about making these changes?

anonymous-matomo-user commented 13 years ago

I patched GeoIP.php based on this file GeoIP-set-location_country.diff, and now there is no logging of access.

Phew....

robocoder commented 13 years ago

If this is a candidate for trunk, I advise renaming it to Geolocation, since "GeoIP" is a trademark. Also, by using a generic, vendor-neutral name, we would be able to add adapters for other geolocation methods/services/products.

mattab commented 13 years ago

I think GeoIP should move to core. This will improve user experience and allow for better Piwik features in the future, such as mapping cities and regions in the world map.

I created a ticket with proposed ideas: #1823

anonymous-matomo-user commented 13 years ago

I tried applying the patch http://issues.piwik.org/attachments/5465/GeoIP-set-location_country.diff but could not get it done.

I used Netbeans before to patch my drupal files but this one seems to have a different file format. Where can I find anything on how to patch under windows?

anonymous-matomo-user commented 13 years ago

I got it done, seems netbeans can not read that patch format and i found GNUWin worked

anonymous-matomo-user commented 13 years ago

I get nearly only unknown users after downloading the version off the description and patching it (I as well run the update php and dropped the tables).

Where do I find the latest version and do I still need toapply the patch ian posted?

This site is pretty confusing to new users. I can find nowhere to get the versions listed.

anonymous-matomo-user commented 13 years ago

Or is there a solution to this

UserCountrycountry UserCountrycontinent

issue I missed??????

anonymous-matomo-user commented 13 years ago

I tried running the script from borwser and command line now. Nothing. ALL of my first user do show up under these lables. I read somewhere above they can not be updated? But what does the script then? If I understand it right The script should query the old data and update the database?! Any help or removing of my confusion would be appreciated :D

anonymous-matomo-user commented 13 years ago

What I need to mention: I did patch before I used GeoIP the first time but I can remember at the second time I ran the script it got 1 lower. I somewhere above read the loop before did only one update - So may there possibly still be a problem?

anonymous-matomo-user commented 13 years ago

if I might add to this. The maxmind db gives a city lookup. This does not work how people think it will. Blocks of IP numbers are sold to to service providers who resell to end users. However, the IP issuing authority assign the city of the ISP address to all the IP numbers. At least that is how it works in the UK. Things may vary in different countries and ISPs don't reallocate city when they sell dedicated IP numbers to end users. The result is that city lookup generally only gives the city of the ISP and not where the visitor is visting from. The ISP can be anywhere in the country and hundreds of miles from where the visitor is based. In other words, city lookup is useless except for giving the location of ISPs. This also means that lat long is useless too since it seems to be based on city lookup. When IPV6 is rolled out and if, and only if, ISPs allocate city to users when they purchase a fixed IP then city lookup may become useful. But many ISPs still use dynamically allocated IPs so it wouldn't work in that case either. In short the concept of providing city and/or lat/long of vistors is fundamentlly flawed.

anonymous-matomo-user commented 13 years ago

Test

anonymous-matomo-user commented 13 years ago

When new visitors appear on the LiveVisitors screen, they initially have the "wrong" flag as with the standard identification by browser settings. Only overnight does GeoIP override the IDs with its correct geodata. How can I change the interval so that the correct identification would be instant or at least happen faster? Great plugin btw.

anonymous-matomo-user commented 13 years ago

UserCountrycountry UserCountrycontinent

Is there any solution for this? Thanks, j

anonymous-matomo-user commented 13 years ago

Replying to tlitody:

if I might add to this. The maxmind db gives a city lookup. This does not work how people think it will. Blocks of IP numbers are sold to to service providers who resell to end users. However, the IP issuing authority assign the city of the ISP address to all the IP numbers. At least that is how it works in the UK. Things may vary in different countries (...)

There is a page on the MaxMind website with a list of the accuracy at country and city level for the GeoLite City database. For the UK they report quite a low accuracy indeed (56% within 25 miles of true location), but for many other countries they do report fairly accurate values (e.g. for the US they report 78%). https://www.maxmind.com/app/geolite_city_accuracy

mattab commented 13 years ago

Published the new GeoIP version for Piwik 1.2 and more.

If you upgrade to Piwik 1.2, you can also apply the following patch rather than downloading the new plugin:

136c136
<                           count(distinct visitor_idcookie) as nb_uniq_visitors,
---
>                           count(distinct idvisitor) as nb_uniq_visitors,

177c176
<       $query = $archiveProcessing->queryConversionsBySegment("location_geoip_continent,location_geoip_country,location_geoip_city");
---
>       $query = $archiveProcessing->queryConversionsByDimension("location_geoip_continent,location_geoip_country,location_geoip_city");
anonymous-matomo-user commented 13 years ago

I am getting an error message when I try to run the update script. I don't know enough php to figure this out, but I'm assuming paths have something to do with it.

$ php /home/me/website.com/piwik/plugins/GeoIP/misc/geoipUpdateRows.php

Warning: require_once(../../../core/testMinimumPhpVersion.php): failed to open stream: No such file or directory in /home/me/website/piwik/plugins/GeoIP/misc/geoipUpdateRows.php on line 23

Fatal error: require_once(): Failed opening required '../../../core/testMinimumPhpVersion.php' (include_path='../../../core:../../../libs:../../../plugins') in /home/me/website/piwik/plugins/GeoIP/misc/geoipUpdateRows.php on line 23

robocoder commented 13 years ago

kip: the script expects to be run from within the same folder

cd /home/me/website.com/piwik/plugins/GeoIP/misc
php geoipUpdateRows.php
anonymous-matomo-user commented 13 years ago

Just installed the new version -- thanks for the fix.

While you're updating, there might be an easy change to make U.S. data a lot more intelligible. Given how hard it is to recognize city names, or distinguish the same city name in multiple states, the two-letter state abbreviations are super-useful:

Will you consider displaying the states for U.S. cities -- e.g. "Springfield, MA" instead of "Springfield"?

anonymous-matomo-user commented 13 years ago

Thanks for piwik 1.2 and this update. Do I have to apply the files in the Updates directory in any way? (07.php, 09.php)?

anonymous-matomo-user commented 13 years ago

I've added " (GeoIP)" to my local en.php and de.php for 'GeoIP_WidgetContinents' and 'GeoIP_WidgetCountries', so that I can distinguish between the normal and the GeoIP widgets. I suggest doing this officially as well. I know that in the forums there have been several requests about GeoIP working at all because people couldn't see a difference.

anonymous-matomo-user commented 13 years ago

One question: I installed the plugin and followed the instructions. GeoIP widget is also shown in piwik. But somehow it doesn't seem to work properly. Piwik shows my IP (79.205.251.97) as "United States" based on my browser locale (which is "en_US"). When I use the geoiplookup command line tool using the same database file "GeoLiteCity.dat" it shows the correct:

$ geoiplookup -f GeoLiteCity.dat 79.205.251.97
GeoIP City Edition, Rev 1: DE, 16, Berlin, N/A, 52.516701, 13.400000, 0, 0

Did piwik cache the result somehow? Should I worry?

Grateful for any suggestions... buzz

robocoder commented 13 years ago

buzz: for performance, geolocation only happens on the first page of a new visit. You either have to wait out the session timeout, or clear your cookies to be treated as a new visitor.

anonymous-matomo-user commented 13 years ago

i downloaded the new zip file and have the following errors

Fatal error: Call to a member function filter() on a non-object in /homepages/39/d319160918/htdocs/i-sitekohchang/analytics/piwik/core/ViewDataTable/HtmlTable/AllColumns.php on line 52

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'visitor_idcookie' in 'field list'

off topic,,, this page (this track ticket) takes ages to load, a lot of comments for 1 page, can you split out the old comments on to a page 2/3..? a log in on the bottom of the page would also be nice :)

anonymous-matomo-user commented 13 years ago

Replying to jekko:

i downloaded the new zip file and have the following (errors on the dashboard not when running geoipUpdateRows.php)

Fatal error: Call to a member function filter() on a non-object in /homepages/39/d319160918/htdocs/i-sitekohchang/analytics/piwik/core/ViewDataTable/HtmlTable/AllColumns.php on line 52

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'visitor_idcookie' in 'field list'

off topic,,, this page (this track ticket) takes ages to load, a lot of comments for 1 page, can you split out the old comments on to a page 2/3..? a log in on the bottom of the page would also be nice :)

anonymous-matomo-user commented 13 years ago

Still got error even i Use the latest GeoIP.2.zip (20.2 KB) for Piwik 1.3

anonymous-matomo-user commented 13 years ago

In order to use GeoIP within Piwik 1.3 I had to add 'require_once('ABSOLUTE_PATH_TO_PIWIK/libs/upgradephp/upgrade.php');" to core/Cookie.php and alter line 373 of core/Config.php, where I changed '_parse_ini_file' to 'parse_ini_file'. This is somehow not the intended behaviour, I guess. A bug?

robocoder commented 13 years ago

fixed in the updated GeoIP.zip package; bumped the version to 0.15

re: page load time of this trac ticket -- this ticket will be closed as soon as we move this plugin into the core distribution; see #1823

robocoder commented 13 years ago

Sorry, repackaged.

anonymous-matomo-user commented 13 years ago

Replying to vipsoft:

Sorry, repackaged.

oh dear!

Fatal error: error traversing database - perhaps it is corrupt? in example.com/analytics/piwik/plugins/GeoIP/libs/geoip.inc on line 422

anonymous-matomo-user commented 13 years ago

Replying to jekko:

Replying to vipsoft:

Sorry, repackaged.

oh dear!

Fatal error: error traversing database - perhaps it is corrupt? in example.com/analytics/piwik/plugins/GeoIP/libs/geoip.inc on line 422 ah ok! the GeoLiteCity.dat did not move across. now its ok but we don't have enough php memory to run it (host will not increase the php memory)

anonymous-matomo-user commented 13 years ago

Replying to jekko:

Replying to jekko:

Replying to vipsoft:

Sorry, repackaged.

oh dear!

Fatal error: error traversing database - perhaps it is corrupt? in example.com/analytics/piwik/plugins/GeoIP/libs/geoip.inc on line 422 ah ok! the GeoLiteCity.dat did not move across. now its ok but we don't have enough php memory to run it (host will not increase the php memory)

its trying to allocate 27 meg, that seems a lot Fatal error: Out of memory (allocated 5242880) (tried to allocate 27629546 bytes)

robocoder commented 13 years ago

Yes, it tries to load the geo database into memory. This is Maxmind's library so it is "by design".

The new geolocation plugin will offer more choices, esp for resource constrained environments.

anonymous-matomo-user commented 13 years ago

Sorry, can't find information on this anywhere. I had GeoIP working, updated Piwik to 1.2 and GeoIP hasn't worked since. Tried to install again with most recent GeoIP.zip but keep getting the error "Unable to load plugin 'GeoIP' because 'var/www/info/piwik/plugins/GeoIP/GeoIP.php' couldn't be found. You can manually uninstall the plugin by removing the line Plugins[] = GeoIP from the Piwik config file."

In piwik config.php.ini the line Plugins[= "GeoIP" has been removed but PluginsInstalled = "GeoIP" remains. (removing it didn't help either)

Any tips/things I should read?

Any help much appreciated

robocoder commented 13 years ago

Remove references to GeoIP from your config.ini.php.

In your plugins/ folder, either remove the GeoIP folder, or re-upload the contents. (note: filenames may be case sensitive)

anonymous-matomo-user commented 13 years ago

Beautiful, thanks very much.

svogt commented 13 years ago

Installed GeoIP v15 into a Piwik 1.3 and the plugin seems to be working (I had a look into the new widget and they show the cities correctly). What currently isn't working is the Visitor-WorldMap which I thought should show the new stats as well. Am I missing something?

robocoder commented 13 years ago

SaschaVogt: that's because we haven't changed the GeoIP plugin to overwrite the standard columns

The GeoIP plugin is in maintenance mode as I'm currently working on its replacement in #1823.