search

matomo-org / matomo

Empowering People Ethically with the leading open source alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites & apps and visualise this data and extract insights. Privacy is built-in. Liberating Web Analytics. Star us on Github? +1. And we love Pull Requests!
https://matomo.org/
GNU General Public License v3.0
19.94k stars 2.66k forks source link

Plugin using GeoIP to accurately detect visitor countries, cities, and continent #5465

Closed mattab closed 10 years ago

mattab commented 16 years ago

GeoIp Plugin

Requirements

You need at least Piwik 0.2.33 to run the GeoIp plugin.

How to install?

How to apply the GeoIp to all your previous visits to fix your previous reports

You can apply GeoIP precise localization on your past data:

Authors

Changelog

Feedback

Please leave a comment if you have any feedback, suggestion, or bug report. Keywords: third-party-plugin

anonymous-matomo-user commented 14 years ago

Attachment: Makes the plugin work when GEO ip is installed as PECL Extension UsePeclExtension.patch

mattab commented 13 years ago

Attachment: 0.18 GeoIP.zip

gka commented 12 years ago

Attachment: I patched GeoIP.php to get it working with the new world map widget. The only thing I changed was to add a column for the GeoIP region code that is returned by the API but wasn't stored in the Piwik database. However the map needs this region code to fill those nice country region maps with colors :) GeoIP.php

anonymous-matomo-user commented 12 years ago

Attachment: 0.18 with minor changes (disabled row evolution, region changes by greg) GeoIP.2.zip

anonymous-matomo-user commented 15 years ago

Thanks for the wonderful work on this plugin, it's really great!

Just wanted to add my little contribution with a little feature I use and thought I would share.

By applying the patch 'patch_GeoIP_0.7_google_map_link' I just uploaded to the original tree it adds links to google maps with the location of cities in the visitor countries dropdown...

mattab commented 15 years ago

mike5464 and rembrand, the plugin should show "Unknown" when the visitor couldn't be located. It will only work well from the upgrade and not fix previous visits. Do you still experience the issue few days after the upgrade?

anonymous-matomo-user commented 15 years ago

@rembrand:

You're using mod_geoip, right? Here's an excerpt from my apache conf:

<IfModule mod_geoip.c> GeoIPEnable On GeoIPEnableUTF8 On GeoIPOutput Env GeoIPDBFile /path/to/GeoLiteCity.dat MemoryCache

for mod_proxy's X-Forwarded-For

    GeoIPScanProxyHeaders On

</IfModule>

Furthermore, check permissions on GeoLiteCity.dat, try chown <apacheuser> /path/to/GeoLiteCity.dat

anonymous-matomo-user commented 15 years ago

wrote a small php script to test the Funktion of Geoip:

<html>
<body>
<?php
 if( !empty($_SERVER['GEOIP_COUNTRY_CODE']) )
                {
                $locationInfo = array();
                $locationInfo['country_code'] = (isset($_SERVER['GEOIP_COUNTRY_CODE'])) ? strtolower($_SERVER['GEOIP_COUNTRY_CODE']) : self::$defaultLocationInfo['country_code'];
                $locationInfo['city'] = (isset($_SERVER['GEOIP_CITY'])) ? utf8_encode($_SERVER['GEOIP_CITY']) : self::$defaultLocationInfo['city'];
                $locationInfo['latitude'] = (isset($_SERVER['GEOIP_LATITUDE'])) ? round($_SERVER['GEOIP_LATITUDE'],4) : self::$defaultLocationInfo['latitude'];
                $locationInfo['longitude'] = (isset($_SERVER['GEOIP_LONGITUDE'])) ? round($_SERVER['GEOIP_LONGITUDE'],4) : self::$defaultLocationInfo['longitude'];

                }

var_dump($locationInfo);
?>
</body>
</html>

Output : array(4) { [ string(2) "de" "city"=> string(7) "Hamburg" [ float(53.55) "longitude"=> float(10) }

Piwik still displays UserCountrycountry for new requests...

anonymous-matomo-user commented 15 years ago

I've extended the GeoIP plugin to track in detail (In the US and Canada) which state or region the visitors are from, by adding an extra column to the database (region) and making minor changes to the code. The region can be query from the maxmind db. I was wondering if this feature can be made public?

mattab commented 15 years ago

duylaiabc, can you please post a screenshot of how the reporting look like?

anonymous-matomo-user commented 15 years ago

it tried to get location_geoip_latitude, location_geoip_longitude from api. I added this two columns to archiveDayAggregateVisits sql query and added

if(!isset($this->interestByCountryAndCity[$row['location_geoip_country']][$row['location_geoip_city']]['loc']))
                $this->interestByCountryAndCity[$row['location_geoip_country']][$row['location_geoip_city']]['loc']=array('lat' => $row['location_geoip_latitude'], 'lon' => $row['location_geoip_longitude']);

But the result is, most of the data have wrong lat and long values. E.g.:

    <row>
        <label>Vienna</label>
        <nb_visits>2</nb_visits>
        <nb_actions>11</nb_actions>
        <max_actions>11</max_actions>
        <sum_visit_length>330</sum_visit_length>

        <bounce_count>0</bounce_count>
        <nb_visits_converted>1</nb_visits_converted>
        <loc>
                <lat>96,4</lat>
                <lon>32,7334</lon>
        </loc>
        <sum_daily_nb_uniq_visitors>2</sum_daily_nb_uniq_visitors>
    </row>
    <row>
        <label>Leonding</label>
        <nb_visits>1</nb_visits>
        <nb_actions>17</nb_actions>

        <max_actions>17</max_actions>
        <sum_visit_length>721</sum_visit_length>
        <bounce_count>0</bounce_count>
        <nb_visits_converted>1</nb_visits_converted>
        <loc>
                <lat>48.2667</lat>

                <lon>14.2500</lon>
        </loc>
        <sum_daily_nb_uniq_visitors>1</sum_daily_nb_uniq_visitors>
    </row>

as you can see, loc data of Leonding is correct, but data of Vienna isn't. Has anybody an idea why?

anonymous-matomo-user commented 15 years ago

for people with limited memory you can replace

$this->geoIpDb = geoip_open($geoIPDataFile, GEOIP_MEMORY_CACHE);

into

$this->geoIpDb = geoip_open($geoIPDataFile, GEOIP_STANDARD);

in GeoIP.php

anonymous-matomo-user commented 15 years ago

Replying to matt:

Replying to mvanlaar:

Is it possible in future version also build a version that can lookup againt a mysql table with the data from citylite?? The file version is very slow under load. And allowing more executing time isn't (imho) a solution.

for more performance the best solution is to install the Mod GEOIP on your apache server. geoip lookups will be lightning fast!

I tested this with the MySQL version of the maxmind database (!http://ipinfodb.com/ip_database.php) and modified the plugin to perform a mysql lookup instead of the binary database and ran the update script several times. A lookup via MySQL takes about 25% longer than the current version and the mysql tables take up about 300 MB. So it's no benefit whatsoever.

anonymous-matomo-user commented 15 years ago

Is this plugin still in development? I would like to use the data from this plugin to show the visitors on Google Maps (or another map service)...

mattab commented 14 years ago

DaSilva, you are welcome to contribute to the plugin. Ideally, you would post a screenshot and a link to your example piwik with the plugin installed so we can have a look and provide Feedback. When your work is done, we can consider including it in the GeoIp plugin.

anonymous-matomo-user commented 14 years ago

Replying to thibaut:

So if anybody has the possibility to craft a GeoIP version (or a working fix) that would run with 0.5.4 I (as many others I'm sure) would greatly appreciate !

I don't know what your problem is, but I have been using this plugins for months, and it is still working with 0.5.4 on my server.

anonymous-matomo-user commented 14 years ago

I have all the time the "unknown" problem and sometimes "UserCountrycountry--" - but most of the time the country-detection works fine. I tried everything :( Same with o.5.1 - so I installad a complete new 0.5.4 with the actual GeoIP from this page and downloaded a new database-file. I followed the instruction one by one. I am sure everything is in the right directory.

Has someone some steps I could try?

anonymous-matomo-user commented 14 years ago

Replying to anamela:

Now I am using (@theodorius123 that should work for you, too!):

if(!defined('PIWIK_INCLUDE_PATH'))
{
  define('PIWIK_INCLUDE_PATH', '../../..');
}
if(!defined('PIWIK_USER_PATH'))
{
  define('PIWIK_USER_PATH', '../../..');
}

Then running the script from the command line SHOULD FUNCTION. yes! Thank you! :)

mmh. this is not realy working for me.

857 rows to process in piwik2_log_visit...

Fatal error: error traversing database - perhaps it is corrupt? in /users/theo/www/piwik/plugins/GeoIP/libs/geoip.inc on line 422

and instead of cities i see all the time "Unknown"

:-(

anonymous-matomo-user commented 14 years ago

After installing everything except this step "For more performance, it is recommanded to install the module apache mod_geoip. Configure your Apache according to:" piwik tracks nothing anymore. After deactivation everythings works fine.

I changed all my previous reports. I've changed the PIWIK_INCLUDE_PATH to my local path in the geoipUpdateRows.php. After that change i could update all my old reports and it works fine. But the problem now is that I couldn't track any new visitor.

anonymous-matomo-user commented 14 years ago

php 5.2 + mysql 5.0.X SLES 10.3 distribution. GeoIP v0.12, piwik 0.5.4 GeoIP.php: updateExistingVisitsWithGeoIpData does not work for me. It only updates one record per each block on limit. Pretty much "while ( $row = $stmt->fetch() )" works for one record only. Changed above line 346 "while ( $row = $stmt->fetch() )" to these two lines. $allRows = $stmt->fetchAll(); foreach ($allRows as $row) works fine after the change.

By the way, using geoipUpdateRows.php script from command line only.

anonymous-matomo-user commented 14 years ago

Is it possible that in the archived data the latitude and longitude will be stored in the database. Would like to use this for a maps plugin.

mattab commented 14 years ago

In Piwik 0.6, there is a backward incompatible API change in the archiving code.

All GeoIP users will need to upgrade to the latest GeoIP pluginafter they upgrade Piwik to 0.6. Piwik will automatically disable the GeoIP plugin during the upgrade to 0.6.

anonymous-matomo-user commented 14 years ago

Hi, I have recently installed the latest version of Piwik .5.5 and it all works fine. I then installed Geoip and again it worked except I get the following problems...

Country (GeoIP)

UserCountrycountry 218

and

Continent (GeoIP)

UserCountrycontinent 218

I have looked through this thread and do not see a solution? Is there one and if so what can be done?

regards,

Blair

anonymous-matomo-user commented 14 years ago

Hi,

I developed some kind of patch to store regions more than city (I find it more practice). You can find it here : Region display patch. Explanation in french (sorry) here : Region display explanations.

Please feel free to tell me if you see something wrong in this patch.

anonymous-matomo-user commented 14 years ago

Replying to e.marguin:

Hi,

I developed some kind of patch to store regions more than city (I find it more practice). You can find it here : Region display patch. Explanation in french (sorry) here : Region display explanations.

Please feel free to tell me if you see something wrong in this patch. Hello,its possible to find any English version of this? Also it's possible to redirect users from mobile/smart phones to a different page? THanks, Mike

anonymous-matomo-user commented 14 years ago

ok i'll just try to translate here the main things (i'm sorry by advance for my poor english)...

The geoip module gives us information about the city of visitors. But often we do not know this city because it's too small. That's why I prefer to display regions, like in the other web analyzers tools.

The patch give upper stores the region name in db instead of the city name.

To install it, just extract it into the plugin/GeoIP/ folder. It do not update old datas. It's technically possible but I didn't need it. If anyone really wants me to write this script, I can.

Eric

anonymous-matomo-user commented 14 years ago

Replying to e.marguin:

ok i'll just try to translate here the main things (i'm sorry by advance for my poor english)...

The geoip module gives us information about the city of visitors. But often we do not know this city because it's too small. That's why I prefer to display regions, like in the other web analyzers tools.

The patch give upper stores the region name in db instead of the city name.

To install it, just extract it into the plugin/GeoIP/ folder. It do not update old datas. It's technically possible but I didn't need it. If anyone really wants me to write this script, I can.

Eric ok,thank you ERIC. I will give it a try.

mattab commented 14 years ago

mike3050, please don't add links to your websites in the trac comments, thank you.

anonymous-matomo-user commented 14 years ago

Hi, I'm happily (== both from app&sys perspectives) running Piwik 0.6.1 + GeoIP using the DB file w/o _modgeoip. Found a minor (at this time) issue: Country numbers don't match with Continent numbers for Europe, where the sum is always minor than the number obtained by summing all the single countries, see below. I'm wondering if this is a bug in the code or in the lookup table. || Country (GeoIP) |||| || Country || Unique visitors || ||Italy || 997 || ||Unknown || 66 || ||United States || 5 || ||Ireland || 3 || ||Great Britain || 3 || ||Poland || 2 || ||Bulgaria || 1 || ||Russia || 1 || ||Switzerland || 1 || ||Germany || 1 ||

|| Continent (GeoIP) |||| || Continent || Unique visitors || ||Europe || 959 || ||Unknown || 66 || ||North America || 5 ||

anonymous-matomo-user commented 14 years ago

Replying to snowdiver:

I experience the problem, that the GeoIP data is always saved as "unknown" in the piwik_log_visit table without any city labels, latitude etc.

I installed the plugin according to your instructions above. GeoIP ver 0.13, piwik ver 0.6.1 GeoIPCity (May 2010)

Is there any hint to fix this problem? Thanks

EDIT: Problem solved. The apache module is installed on my server, but doesn't provide enough geo-information. Switching the module-search off fixed this problem!

anonymous-matomo-user commented 14 years ago

Replying to snowdiver:

EDIT: Problem solved. The apache module is installed on my server, but doesn't provide enough geo-information. Switching the module-search off fixed this problem!

sorry, i am not that fit with all these server, php, etc stuff. i am using piwik and geoip and have the same "unknown" problem. but i dont have an own server, my homepage is on a freehoster-server (bplaced.net). is it possible to do what you did when i am using a free-hoster??

thanks! theo

anonymous-matomo-user commented 14 years ago

Replying to theodorius123:

Replying to snowdiver:

EDIT: Problem solved. The apache module is installed on my server, but doesn't provide enough geo-information. Switching the module-search off fixed this problem!

sorry, i am not that fit with all these server, php, etc stuff. i am using piwik and geoip and have the same "unknown" problem. but i dont have an own server, my homepage is on a freehoster-server (bplaced.net). is it possible to do what you did when i am using a free-hoster??

thanks! theo

My homepage is also hosted on bplaced.net servers, so we met the same problem. It is just a small modification of the GeoIP.php file. Just remove or comment lines 230-234 to force the plugin to use the GeoLiteCity.dat file instead of the Apache Module, which in the bplaced.net case does not provide information about the city of a visitor.

anonymous-matomo-user commented 14 years ago

The most frequent "city" in my stats is "unknown". Any suggestion for a better geoip database? I'm concerned only with Brazil, FWIW.

mattab commented 14 years ago

All users experimenting issues with the geoipUpdateRows.php script, we have fixed a few bugs reported by users in this thread. Check out the new GeoIP 0.14 version.

If you still have issues, please report here, we'll do our best to fix it.

Users that are seeing a lot of 'unknown' countries and cities, this is probably due to the low coverage of the Free IP to country database provided by MaxMind. If anyone knows a better DB, let us know! You can alternatively buy the MaxMind pro DB, but I haven't tested how accurate this DB is.

anonymous-matomo-user commented 14 years ago

How do I know if a database is in the file format for GeoIP?

anonymous-matomo-user commented 14 years ago

I seem to have some problem with the geoipUpdateRows.php script. What I can see, nothing happens when I run the script, all older data is just NULL, newer data from the same IP is however marked correctly.

anonymous-matomo-user commented 14 years ago

btw: could it be that the geo-ip colums stay in the db after plugin deactivation? Is this bahavior wanted?

mikeSimonson commented 14 years ago

With the last version, i still get this error.

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator, postmaster@www.climb2climb.be and inform them of the time the error occurred, and anything you might have done that may have caused the error.

More information about this error may be available in the server error log.

anonymous-matomo-user commented 14 years ago

I'm trying to run the latest version of this plugin on Piwik 0.9 and all I seem to be getting is UserCountrycountry as Blair noted in Comment #101. Has anyone found a solution for this?

anonymous-matomo-user commented 14 years ago

I had a problem of exit without error and no log of my visits, i followed this: Replying to marcello.ceschia:

for people with limited memory you can replace

$this->geoIpDb = geoip_open($geoIPDataFile, GEOIP_MEMORY_CACHE);

into

$this->geoIpDb = geoip_open($geoIPDataFile, GEOIP_STANDARD);

in GeoIP.php

and it worked, on a dedicated server with 2Go of ram ...

mattab commented 14 years ago

From forum post GeoIP just show true country in "Locations & Provider" and it doesn't show true country in "Visitor Log", "world map" and "Live Visitors!"

anonymous-matomo-user commented 14 years ago

If the GeoIP plugin writes to "the existing country & continent columns", then what should it write when the country is unknown? In my experience (YMMV) that's the second most frequent "country". The IP is already stored by Piwik, so if the geolocations are to be stored by GeoIP, each database should have its own columns in the database. Each database has its own limitation, and mixing information from more than one database in a single column would impair the user from using the correct precautions.

robocoder commented 14 years ago

As you say, each database has its limitations. Each also has different operating characteristics (e.g., system requirements, memory requirement, disk space requierments, and lookup time). I proposed that users be able to pick and choose which one to use, if any.

Users generally expect/assume the existing country & continent columns to be geolocated. (We have a FAQ to explain why it isn't.) What we can do is that when the country/continent is unknown, the plugin fallback to what we do now when GeoIP isn't installed/activated, i.e., guess the country from the Accept-Language header; and if that fails, then write 'xx'.

anonymous-matomo-user commented 14 years ago

I agree that the existing country & continent columns should be geolocated when possible, with a fallback to the no-plugin behavior on "unknown" results. Would it be possible to fall back to the actual "faux geolocation" code that populates those columns now so there isn't code duplication in the plugin?

I can't think of a case where this would worsen the data in the existing columns. It's going to be either an improvement (when there is GeoIP data for the IP, which "outranks" a guess from the language header) or no change (when there is fallback to the language header code).

I'd also be interested to know if the paid version of the GeoIP database cuts down significantly on the number of unknowns. I do get valuable information from the 30% or so of my traffic that shows up in GeoIP as "unknown" - since I know the GeoIP database coverage of USA addresses is very high, I can reasonably assume that most of that traffic is from outside the US. It does look like the language-header method guesses USA for a lot of non-US addresses.. and for all my traffic from India.

anonymous-matomo-user commented 14 years ago

Turned out to be a very simple patch. I think this is low risk - it leaves location_country alone unless GeoIP knows what the country is. Ran it against the historical visits on my end and the Core and GeoIP "Visitor Countries" widgets are now much closer. No "unknowns" showed up in the Core widget, but India, which had been completely absent, is there now. (Pretty much everyone in India has their browser language set to English/US, apparently..)

I would appreciate feedback/review, this is my first submitted Piwik patch.

lehrer commented 14 years ago

Hi,

I am using piwik 1.0.

I get the following error (same as reported above a few times): UserCountrycountry

How can I visualize the countries and continetns instead of UserCountrycountry and UserCountrycontinent respectively?

GL

anonymous-matomo-user commented 14 years ago

I have the same error

anonymous-matomo-user commented 14 years ago

I've been using this plugin for two weeks. With American cities, it would be help me to display them as "Springfield, MA" rather than just "Springfield". Often the city name won't mean much to me on its own but the state will.

(Showing postal abbrevations for states and provinces in other countries would be welcome too, but I wouldn't be a good test user for that.)

anonymous-matomo-user commented 14 years ago

GeoIP is activated but only shows the correct country entered visitors> location.

But in the plugins UserCountry, UserCountryMap, live, continents, are the wrong country and continent.

I am from Ecuador South America continent, Spanish, shows such as Spain, mainland Europe Spanish.

I searched for information but can not find clear and specific answer to this problem.

How I can fix it?

anonymous-matomo-user commented 14 years ago

If I'm correct in Piwik demo is not activated or installed GeoIP, however UserCountryMap plugin register, live, UserCountry, etc, Ecuador South America visits.

What I can do to register the countries and continents as they recorded a demo of Piwik?

Esteban