search

matomo-org / matomo

Empowering People Ethically with the leading open source alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites & apps and visualise this data and extract insights. Privacy is built-in. Liberating Web Analytics. Star us on Github? +1. And we love Pull Requests!
https://matomo.org/
GNU General Public License v3.0
19.67k stars 2.62k forks source link

GeoIPAutoUpdater fails with "failed to unzip '/var/www/stat/misc/GeoIPCity.dat.dat.gz' #9544

Closed mattab closed 5 years ago

mattab commented 8 years ago

GeoIPAutoUpdater: failed to unzip '/var/www/stat/misc/GeoIPCity.dat.dat.gz' after downloading 'http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz': The downloaded file is not a valid GeoIP database. Please re-check the URL or download the file manually.

As reported in http://forum.piwik.org/t/geoipautoupdater-error/16143

mattab commented 8 years ago

Notes

Full error trace is:

ERROR [2016-01-06 00:56:32] /home/piwik-demo/www/demo.piwik.org/plugins/UserCountry/GeoIPAutoUpdater.php(161): GeoIPAutoUpdater: failed to unzip '/home/piwik-demo/www/demo.piwik.org/misc/GeoIPCity.dat.dat.gz' after downloading 'http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz': The downloaded file is not a valid GeoIP database. Please re-check the URL or download the file manually.
#0 /home/piwik-demo/www/demo.piwik.org/plugins/UserCountry/GeoIPAutoUpdater.php(96): Piwik\Plugins\UserCountry\GeoIPAutoUpdater->downloadFile('loc', 'http://geolite....')
#1 [internal function]: Piwik\Plugins\UserCountry\GeoIPAutoUpdater->update(NULL)
#2 /home/piwik-demo/www/demo.piwik.org/core/Scheduler/Scheduler.php(227): call_user_func(Array, NULL)
#3 /home/piwik-demo/www/demo.piwik.org/core/Scheduler/Scheduler.php(124): Piwik\Scheduler\Scheduler->executeTask(Object(Piwik\Plugins\UserCountry\GeoIPAutoUpdater))
#4 /home/piwik-demo/www/demo.piwik.org/plugins/CoreAdminHome/API.php(55): Piwik\Scheduler\Scheduler->run()
#5 /home/piwik-demo/www/demo.piwik.org/core/CronArchive.php(529): Piwik\Plugins\CoreAdminHome\API->runScheduledTasks()
#6 /home/piwik-demo/www/demo.piwik.org/core/CronArchive.php(283): Piwik\CronArchive->runScheduledTasks()
#7 /home/piwik-demo/www/demo.piwik.org/core/Access.php(466): Piwik\CronArchive->Piwik\{closure}()
#8 /home/piwik-demo/www/demo.piwik.org/core/CronArchive.php(285): Piwik\Access::doAsSuperUser(Object(Closure))
#9 /home/piwik-demo/www/demo.piwik.org/plugins/CoreConsole/Commands/CoreArchiver.php(27): Piwik\CronArchive->main()
#10 /home/piwik-demo/www/demo.piwik.org/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(257): Piwik\Plugins\CoreConsole\Commands\CoreArchiver->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /home/piwik-demo/www/demo.piwik.org/vendor/symfony/console/Symfony/Component/Console/Application.php(874): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /home/piwik-demo/www/demo.piwik.org/vendor/symfony/console/Symfony/Component/Console/Application.php(195): Symfony\Component\Console\Application->doRunCommand(Object(Piwik\Plugins\CoreConsole\Commands\CoreArchiver), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 [internal function]: Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /home/piwik-demo/www/demo.piwik.org/core/Console.php(79): call_user_func(Array, Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /home/piwik-demo/www/demo.piwik.org/core/Access.php(466): Piwik\Console->Piwik\{closure}()
#16 /home/piwik-demo/www/demo.piwik.org/core/Console.php(80): Piwik\Access::doAsSuperUser(Object(Closure))
#17 /home/piwik-demo/www/demo.piwik.org/vendor/symfony/console/Symfony/Component/Console/Application.php(126): Piwik\Console->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /home/piwik-demo/www/demo.piwik.org/misc/cron/archive.php(57): Symfony\Component\Console\Application->run()
#19 {main}
mattab commented 8 years ago

Adding to 2.16.0 - @tsteur @sgiehl maybe you could take a look at this one? It started occuring in November and seems to be experienced by all users.

tsteur commented 8 years ago

FYI: geoip.loc_db_url in option table in DB says http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz on demo. Had a quick look but code is almost impossible to understand so needs proper investigating or refactoring.

hpvd commented 8 years ago

in our installation (2.16b3) in backend there is no error message visible. Looks "working ok" => is it ok in this installation (and does not appear everywhere)? => or should one add a error massage to report if there is a problem? please see attachement

2016-01-18_23h06_34

clauso commented 8 years ago

I have the same error today (I think I have seen it in the past):

ERROR [2016-02-03 00:05:11] /var/www/piwik/plugins/UserCountry/GeoIPAutoUpdater.php(161): GeoIPAutoUpdater: failed to unzip '/var/www/piwik/misc/GeoIPCity.dat.dat.gz' after downloading 'http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz': The downloaded file is not a valid GeoIP database. Please re-check the URL or download the file manually.

We are running piwik 2.15.0

I have this md5sum for the downloaded file:

$> md5sum /var/www/piwik/misc/GeoLiteCity.dat
97752fa1e1407d9c2c48008a4b9428c1  /var/www/piwik/misc/GeoLiteCity.dat
nd2s commented 8 years ago

Fails here as well. v2.16.0 on FreeBSD.

clauso commented 8 years ago

Still get those mails about failures so I tried to look into it. I found out that I can trigger an update run manually by calling

php /var/www/piwik/console scheduled-tasks:run --force

A maybe interesting thing is that here I get an additional error that was not in the mails, see line 2:

INFO [2016-07-13 04:59:50] GeoIPAutoUpdater: successfully downloaded 'http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz'
INFO [2016-07-13 04:59:50] GeoIPAutoUpdater: Encountered PHP error when testing newly downloaded GeoIP database: 2: include(): Failed opening '/var/www/piwik/tmp/cache/tracker/piwikcache_Translations-en-e21bf04aeec884480f77854677fd2bf5fbc090f3.php' for inclusion (include_path='/var/www/piwik/vendor/pear/pear_exception:/var/www/piwik/vendor/pear/console_getopt:/var/www/piwik/vendor/pear/pear-core-minimal/src:/var/www/piwik/vendor/pear/archive_tar:.:/usr/share/php') on line 112 of /var/www/piwik/vendor/doctrine/cache/lib/Doctrine/Common/Cache/PhpFileCache.php.
ERROR [2016-07-13 04:59:50] /var/www/piwik/plugins/UserCountry/GeoIPAutoUpdater.php(161): GeoIPAutoUpdater: failed to unzip '/var/www/piwik/misc/GeoIPCity.dat.dat.gz' after downloading 'http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz': The downloaded file is not a valid GeoIP database. Please re-check the URL or download the file manually.

I currently do not have the piwik knowledge and time to investigate further but maybe this give the right person a hint?

jonroberts-anr commented 7 years ago

I blame Cloudflare. On our server we tried to wget and curl the url manually. wget gives me this: Connecting to geolite.maxmind.com (geolite.maxmind.com)|104.16.38.47|:80... connected. HTTP request sent, awaiting response... 503 Service Temporarily Unavailable 2017-05-14 00:09:56 ERROR 503: Service Temporarily Unavailable.

curl gives me this: <!DOCTYPE HTML>

Just a moment...
DDoS protection by Cloudflare
Ray ID: 35e99544e3061049
robertoschwald commented 6 years ago

Any news on this? Facing this problem since a long time

JonTheWong commented 5 years ago

I've been recently getting the same error. My current error output is for GeoLite2 but I've tried legacy with no success. The URL is giving me a 404 on legacy. Geo2 doesn't work because it doesn't contain a .dat file.

System Check "Huzzah! There are no problems with your Matomo setup. Give yourself a pat on the back."

Matomo 3.7.0 Current.

ERROR [2019-01-13 11:49:21] 6701  /home/user/public_html/analytics.example.com/plugins/UserCountry/GeoIPAutoUpdater.php(167): GeoIPAutoUpdater: failed to unzip '/home/user/public_html/analytics.example.com/misc/GeoIPCity.dat.tar.gz' after downloading 'https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz': Cannot find GeoIPCity.dat file in tar archive '/home/user/public_html/analytics.example.com/misc/GeoIPCity.dat.tar.gz'!
#0 /home/user/public_html/analytics.example.com/plugins/UserCountry/GeoIPAutoUpdater.php(102): Piwik\Plugins\UserCountry\GeoIPAutoUpdater->downloadFile('loc', 'https://geolite...')
#1 [internal function]: Piwik\Plugins\UserCountry\GeoIPAutoUpdater->update(NULL)
#2 /home/user/public_html/analytics.example.com/core/Scheduler/Scheduler.php(246): call_user_func(Array, NULL)
#3 /home/user/public_html/analytics.example.com/core/Scheduler/Scheduler.php(136): Piwik\Scheduler\Scheduler->executeTask(Object(Piwik\Plugins\UserCountry\GeoIPAutoUpdater))
#4 /home/user/public_html/analytics.example.com/plugins/CoreAdminHome/API.php(58): Piwik\Scheduler\Scheduler->run()
#5 /home/user/public_html/analytics.example.com/core/CronArchive.php(612): Piwik\Plugins\CoreAdminHome\API->runScheduledTasks()
#6 /home/user/public_html/analytics.example.com/core/CronArchive.php(317): Piwik\CronArchive->runScheduledTasks()
#7 /home/user/public_html/analytics.example.com/core/Access.php(626): Piwik\CronArchive->Piwik\{closure}()
#8 /home/user/public_html/analytics.example.com/core/CronArchive.php(319): Piwik\Access::doAsSuperUser(Object(Closure))
#9 /home/user/public_html/analytics.example.com/plugins/CoreConsole/Commands/CoreArchiver.php(27): Piwik\CronArchive->main()
#10 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(257): Piwik\Plugins\CoreConsole\Commands\CoreArchiver->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Application.php(874): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Application.php(195): Symfony\Component\Console\Application->doRunCommand(Object(Piwik\Plugins\CoreConsole\Commands\CoreArchiver), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 [internal function]: Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /home/user/public_html/analytics.example.com/core/Console.php(90): call_user_func(Array, Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /home/user/public_html/analytics.example.com/core/Access.php(626): Piwik\Console->Piwik\{closure}()
#16 /home/user/public_html/analytics.example.com/core/Console.php(91): Piwik\Access::doAsSuperUser(Object(Closure))
#17 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Application.php(126): Piwik\Console->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /home/user/public_html/analytics.example.com/console(27): Symfony\Component\Console\Application->run()
#19 {main}

Found the original log.

ERROR [2019-01-09 11:45:50] 12232  /home/user/public_html/analytics.example.com/plugins/UserCountry/GeoIPAutoUpdater.php(167): GeoIPAutoUpdater: failed to unzip '/home/user/public_html/analytics.example.com/misc/GeoIPCity.dat.dat.gz' after downloading 'http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz': The downloaded file is not a valid GeoIP database. Please re-check the URL or download the file manually.
#0 /home/user/public_html/analytics.example.com/plugins/UserCountry/GeoIPAutoUpdater.php(102): Piwik\Plugins\UserCountry\GeoIPAutoUpdater->downloadFile('loc', 'http://geolite....')
#1 [internal function]: Piwik\Plugins\UserCountry\GeoIPAutoUpdater->update(NULL)
#2 /home/user/public_html/analytics.example.com/core/Scheduler/Scheduler.php(246): call_user_func(Array, NULL)
#3 /home/user/public_html/analytics.example.com/core/Scheduler/Scheduler.php(136): Piwik\Scheduler\Scheduler->executeTask(Object(Piwik\Plugins\UserCountry\GeoIPAutoUpdater))
#4 /home/user/public_html/analytics.example.com/plugins/CoreAdminHome/API.php(58): Piwik\Scheduler\Scheduler->run()
#5 /home/user/public_html/analytics.example.com/core/CronArchive.php(612): Piwik\Plugins\CoreAdminHome\API->runScheduledTasks()
#6 /home/user/public_html/analytics.example.com/core/CronArchive.php(317): Piwik\CronArchive->runScheduledTasks()
#7 /home/user/public_html/analytics.example.com/core/Access.php(626): Piwik\CronArchive->Piwik\{closure}()
#8 /home/user/public_html/analytics.example.com/core/CronArchive.php(319): Piwik\Access::doAsSuperUser(Object(Closure))
#9 /home/user/public_html/analytics.example.com/plugins/CoreConsole/Commands/CoreArchiver.php(27): Piwik\CronArchive->main()
#10 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Command/Command.php(257): Piwik\Plugins\CoreConsole\Commands\CoreArchiver->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#11 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Application.php(874): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#12 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Application.php(195): Symfony\Component\Console\Application->doRunCommand(Object(Piwik\Plugins\CoreConsole\Commands\CoreArchiver), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#13 [internal function]: Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#14 /home/user/public_html/analytics.example.com/core/Console.php(90): call_user_func(Array, Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#15 /home/user/public_html/analytics.example.com/core/Access.php(626): Piwik\Console->Piwik\{closure}()
#16 /home/user/public_html/analytics.example.com/core/Console.php(91): Piwik\Access::doAsSuperUser(Object(Closure))
#17 /home/user/public_html/analytics.example.com/vendor/symfony/console/Symfony/Component/Console/Application.php(126): Piwik\Console->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
#18 /home/user/public_html/analytics.example.com/console(27): Symfony\Component\Console\Application->run()
#19 {main}
OliverTempel commented 5 years ago

Same Error Here.

OliverTempel commented 5 years ago

MaxMind has changed downloads. New Files has MaxMind Database Format, so no .dat file is included in download.

sgiehl commented 5 years ago

GeoIP Legacy Lite has been announced to be discontinued about a year ago. Since two weeks there are no downloads available anymore. If you wish to use a free database you need to switch to GeoIP 2. To do this you may need to activate the GeoIp2 plugin in Matomo and configured it in the location provider settings.

will close this issue as we won't/can't fix that anymore

refs #13957