When misc/GeoLiteCity.dat file is missing, this can break Piwik

[mattab]

Right now, demo.piwik.org shows "This connection was reset" error. It's completely broken and not accessible. Looking at errors logs we can see continuously these errors added:

[Wed Aug 05 15:19:24 2015] [error] [mod_geoip]: Error while opening data file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Aug 05 15:19:25 2015] [notice] child pid 510957 exit signal Segmentation fault (11)
Error Opening file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat

Suggested solution:

• when the /misc/GeoLiteCity.dat file is corrupted, Piwik / PHP process should not crash
• can we add a test that simulates a corrupt /misc/GeoLiteCity.dat and check that Piwik tracking + UI still work?

[mattab]

It breaks Tracker too! eg. http://demo.piwik.org/piwik.php?redirecturl=http%3A%2F%2Fplugins.piwik.org%2FCustomOptOut&c_i=click...... does not redirect.

Looking on the server, the file does not actually exist.

$ ls -la /home/piwik-demo/www/demo.piwik.org/misc/Geo*
-rw-r--r-- 1 piwik-demo piwik-demo 19237666 2012-12-21 10:57 /home/piwik-demo/www/demo.piwik.org/misc/GeoIPCity.dat.broken
-rw-r--r-- 1 piwik-demo piwik-demo  2918213 2014-11-05 18:35 /home/piwik-demo/www/demo.piwik.org/misc/GeoIPISP.dat
-rw-r--r-- 1 piwik-demo piwik-demo 32646602 2014-11-05 18:35 /home/piwik-demo/www/demo.piwik.org/misc/GeoIPOrg.dat
-rw-r--r-- 1 piwik-demo piwik-demo 19237666 2012-12-21 10:57 /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat.broken

I'm wondering why Piwik tries to load the file when it does not even exist, maybe this is the bug?

[mattab]

FYI: to workaround this issue I had to re-download the geoip database:

$ wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
$ gunzip GeoLiteCity.dat.gz
$ ls GeoLiteCity.dat -la
-rw-rw-r-- 1 piwik-demo piwik-demo 15418580 2015-08-05 05:46 GeoLiteCity.dat

[tsteur]

I tried to reproduce by removing the file misc/GeoIP.dat and by corrupting this file (removing lots of content everywhere, replacing it with other files etc). I cannot reproduce it neither in Tracker nor in Piwik. Also tried Default Provider, PHP provider and PECL provider. Even restarted apache etc after each change.

Do you still have that corrupt file? Tried http://demo.piwik.org/misc/GeoIPCity.dat.broken but doesn't work. Which provider is used?

[mattab]

unfortunately we cannot reproduce the issue yet.

If you experience this issue, please comment here!

[mattab]

Again got the issue today. Demo.piwik.org is down, error log shows:

Error Opening file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:50 2015] [error] [mod_geoip]: Error while opening data file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:51 2015] [notice] child pid 3040200 exit signal Segmentation fault (11)
Error Opening file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:52 2015] [error] [mod_geoip]: Error while opening data file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:53 2015] [notice] child pid 3040305 exit signal Segmentation fault (11)
Error Opening file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:54 2015] [error] [mod_geoip]: Error while opening data file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:55 2015] [notice] child pid 3040425 exit signal Segmentation fault (11)
Error Opening file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:56 2015] [error] [mod_geoip]: Error while opening data file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:57 2015] [notice] child pid 3040529 exit signal Segmentation fault (11)
Error Opening file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:58 2015] [error] [mod_geoip]: Error while opening data file /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
[Wed Sep 02 08:19:59 2015] [notice] child pid 3040641 exit signal Segmentation fault (11)

the file is that is being opened does not even exist:

piwik-demo@piwik:~/www/demo.piwik.org$ ls -la /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat
ls: cannot access /home/piwik-demo/www/demo.piwik.org/misc/GeoLiteCity.dat: No such file or directory

It's worrying that when this file is not even there, Piwik is broken in this way. let's investigate!

[tsteur]

I investigated and couldn't reproduce that time. Maybe someone else can

[sgiehl]

Do we have mod_geoip for apache on demo.piwik.org? Maybe it crashes if the configured GeoIPDBFile is not found?

[mattab]

Do we have mod_geoip for apache on demo.piwik.org?

Yes mod_geoip for apache is activated on demo

Maybe it crashes if the configured GeoIPDBFile is not found?

Maybe indeed, if you can reproduce let us know!

[sgiehl]

Haven't installed mod_geoip yet, but I'll try to reproduce later...

[sgiehl]

I've just installed mod_geoip on my development box. Removing or corrupting does produce the " Error while opening data file" but everything is still working - no segfault. Which version of apache/mod_geoip is installed?

[mattab]

Apache 2.2

$ apt-cache show libapache2-mod-geoip | grep Version
Version: 1.1.8-2

[stijnstijn]

I am having this exact issue. Running a Piwik install without problems, followed the steps in the FAQ for installing Geolocation via PECL, and now the config page for geolocation won't load and that segmentation fault shows up in the log.

I haven't really been able to find out more about this and the workaround of redownloading the database didn't work for me, but I'll be happy to do anything that may help you debug this.

Interestingly when I comment out the geoip.custom_directory directive in php.ini, everything works again and PECL geolocation works as well. The config page says the directive is empty but I suppose Piwik found the way to the database file on its own.

[raj45]

The GeoIPCity.dat was renamed by the system to GeoIPCity.dat.broken, and geotracking stopped working, I don't know when. My Piwik-installation worked fine, though.

I restored a fresh copy of GeoIPCity.dat to the /misc folder today, and everything seems to be working again. I am using the GeoIP (Apache) service.

[raj45]

I got the below error by mail, and the location no longer works. Also, GeoIPCity.dat in the /misc folder has been renamed to GeoIPCity.dat.broken ...

ERROR [2016-02-03 00:28:55] GeoIPAutoUpdater: Encountered PHP error when performing redundant tests on GeoIP loc database: 2: include(): Failed opening '/data/www/website/tmp/cache/tracker/piwikcache_Translations-en-ebd64ae248b2308a68b47baeadaf0cc549a66c38.php' for inclusion (include_path='/data/www/website/vendor/pear/pear_exception:/data/www/website/vendor/pear/console_getopt:/data/www/website/vendor/pear/pear-core-minimal/src:/data/www/website/vendor/pear/archive_tar:.:/usr/share/php:/usr/share/pear') on line 112 of /data/www/website/vendor/doctrine/cache/lib/Doctrine/Common/Cache/PhpFileCache.php.

[mattab]

@sgiehl Maybe you have a chance to look into this issue? we also experience it on demo2, it would be awesome to address it in our LTS :+1:

[sgiehl]

I've retried to reproduce, but I'm not able to reproduce the segfault. Maybe it only occurs with a special combinations of os/php/pecl versions.

We maybe should try to reduce to possibility that those files are even missing. If I've seen that correctly, the GeoIPUpdater tries to download and update those files. If a downloaded file doesn't work it will be renambed to *.broken. Why don't we keep a backup of the file that was used before, and reuse that one if the update fails?

[sballinone]

Same issue for me. If I have a GeoIPCity.dat located in the misc directory, it breaks Piwik. If not, Piwik runs. Ubuntu 16.04, Matomo 3.3.0, php7, geoip1.1.1

[sgiehl]

refs #13957

[sgiehl]

I will close this issue now. GeoIP was deprecated by maxmind, and thus using the Pecl extension doesn't make much sense anymore, as it doesn't support GeoIP2 databases.