search

matomo-org / matomo

Empowering People Ethically with the leading open source alternative to Google Analytics that gives you full control over your data. Matomo lets you easily collect data from websites & apps and visualise this data and extract insights. Privacy is built-in. Liberating Web Analytics. Star us on Github? +1. And we love Pull Requests!
https://matomo.org/
GNU General Public License v3.0
19.8k stars 2.64k forks source link

GeoIPAutoUpdater fails updating geoip2 database with .mmdb file ending #11049

Closed slawa-dev closed 6 years ago

slawa-dev commented 7 years ago

Error: Encountered unsupported archive type 'mmdb.gz'

MaxMind geoip2 uses new format for goip database with .mmdb file ending. e.g. GeoLite2-City.mmdb http://dev.maxmind.com/geoip/geoip2/geolite2/

The download url for the file is: http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz

cron/archive.php generates an error when auto updating.

INFO CoreAdminHome[2016-12-20 01:13:21] [d3190] Scheduler: executing task Piwik\Plugins\UserCountry\GeoIPAutoUpdater.update...
ERROR UserCountry[2016-12-20 01:13:21] [d3190] /var/www/example.com/plugins/UserCountry/GeoIPAutoUpdater.php(489): Encountered unsupported archive type 'mmdb.gz'.
  #0 /var/www/example.com/plugins/UserCountry/GeoIPAutoUpdater.php(471): Piwik\Plugins\UserCountry\GeoIPAutoUpdater::checkForSupportedArchiveType('mmdb.gz')
  #1 /var/www/example.com/plugins/UserCountry/GeoIPAutoUpdater.php(134): Piwik\Plugins\UserCountry\GeoIPAutoUpdater::getGeoIPUrlExtension('http://geolite....')
  #2 /var/www/example.com/plugins/UserCountry/GeoIPAutoUpdater.php(96): Piwik\Plugins\UserCountry\GeoIPAutoUpdater->downloadFile('loc', 'http://geolite....')
  #3 [internal function]: Piwik\Plugins\UserCountry\GeoIPAutoUpdater->update(NULL)
  #4 /var/www/example.com/core/Scheduler/Scheduler.php(227): call_user_func(Array, NULL)
  #5 /var/www/example.com/core/Scheduler/Scheduler.php(124): Piwik\Scheduler\Scheduler->executeTask(Object(Piwik\Plugins\UserCountry\GeoIPAutoUpdater))
  #6 /var/www/example.com/plugins/CoreAdminHome/API.php(58): Piwik\Scheduler\Scheduler->run()
  #7 /var/www/example.com/core/CronArchive.php(536): Piwik\Plugins\CoreAdminHome\API->runScheduledTasks()
  #8 /var/www/example.com/core/CronArchive.php(290): Piwik\CronArchive->runScheduledTasks()
  #9 /var/www/example.com/core/Access.php(469): Piwik\CronArchive->Piwik\{closure}()
  #10 /var/www/example.com/core/CronArchive.php(292): Piwik\Access::doAsSuperUser(Object(Closure))
  #11 /var/www/example.com/plugins/CoreAdminHome/API.php(179): Piwik\CronArchive->main()
  #12 [internal function]: Piwik\Plugins\CoreAdminHome\API->runCronArchiving()
  #13 /var/www/example.com/core/API/Proxy.php(219): call_user_func_array(Array, Array)
  #14 /var/www/example.com/core/API/Request.php(236): Piwik\API\Proxy->call('\\Piwik\\Plugins\\...', 'runCronArchivin...', Array)
  #15 /var/www/example.com/plugins/API/Controller.php(40): Piwik\API\Request->process()
  #16 [internal function]: Piwik\Plugins\API\Controller->index()
  #17 /var/www/example.com/core/FrontController.php(518): call_user_func_array(Array, Array)
  #18 /var/www/example.com/core/FrontController.php(122): Piwik\FrontController->doDispatch('API', false, Array)
  #19 /var/www/example.com/core/dispatch.php(34): Piwik\FrontController->dispatch()
  #20 /var/www/example.com/index.php(27): require_once('/var/www/exampl...')
  #21 /var/www/example.com/misc/cron/archive.php(70): require_once('/var/www/exampl...')
  #22 {main}
INFO CoreAdminHome[2016-12-20 01:13:21] [d3190] Scheduler: finished. Time elapsed: 0.013s
mattab commented 7 years ago

@s1awa thanks for the report. i've just added the extension to our white list in: https://github.com/piwik/piwik/pull/11081/files

Could you check if it works after updating the PR, since I haven't tested?

slawa-dev commented 7 years ago

@mattab

The downloaded file is not a valid GeoIP database. Please re-check the URL or download the file manually.

There are more checks for downloaded GeoIP Databases.

INFO UserCountry[2016-12-26 01:22:59] [5fd6a] GeoIPAutoUpdater: successfully downloaded 'http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz'
INFO UserCountry[2016-12-26 01:23:00] [5fd6a] GeoIPAutoUpdater: Encountered PHP error when testing newly downloaded GeoIP database: 8: Undefined offset: -16776960 on line 1705 of /home/www/analytics/libs/MaxMindGeoIP/geoip.inc.
ERROR UserCountry[2016-12-26 01:23:00] [5fd6a] /home/www/analytics/plugins/UserCountry/GeoIPAutoUpdater.php(161): GeoIPAutoUpdater: failed to unzip '/home/www/analytics/misc/GeoIPCity.dat.mmdb.gz' after downloading 'http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz': The downloaded file is not a valid GeoIP database. Please re-check the URL or download the file manually.
  #0 /home/www/analytics/plugins/UserCountry/GeoIPAutoUpdater.php(96): Piwik\Plugins\UserCountry\GeoIPAutoUpdater->downloadFile('loc', 'http://geolite....')
  #1 [internal function]: Piwik\Plugins\UserCountry\GeoIPAutoUpdater->update(NULL)
  #2 /home/www/analytics/core/Scheduler/Scheduler.php(227): call_user_func(Array, NULL)
  #3 /home/www/analytics/core/Scheduler/Scheduler.php(124): Piwik\Scheduler\Scheduler->executeTask(Object(Piwik\Plugins\UserCountry\GeoIPAutoUpdater))
  #4 /home/www/analytics/plugins/CoreAdminHome/API.php(58): Piwik\Scheduler\Scheduler->run()
  #5 /home/www/analytics/core/CronArchive.php(536): Piwik\Plugins\CoreAdminHome\API->runScheduledTasks()
  #6 /home/www/analytics/core/CronArchive.php(290): Piwik\CronArchive->runScheduledTasks()
  #7 /home/www/analytics/core/Access.php(469): Piwik\CronArchive->Piwik\{closure}()
  #8 /home/www/analytics/core/CronArchive.php(292): Piwik\Access::doAsSuperUser(Object(Closure))
  #9 /home/www/analytics/plugins/CoreAdminHome/API.php(179): Piwik\CronArchive->main()
  #10 [internal function]: Piwik\Plugins\CoreAdminHome\API->runCronArchiving()
  #11 /home/www/analytics/core/API/Proxy.php(219): call_user_func_array(Array, Array)
  #12 /home/www/analytics/core/API/Request.php(236): Piwik\API\Proxy->call('\\Piwik\\Plugins\\...', 'runCronArchivin...', Array)
  #13 /home/www/analytics/plugins/API/Controller.php(40): Piwik\API\Request->process()
  #14 [internal function]: Piwik\Plugins\API\Controller->index()
  #15 /home/www/analytics/core/FrontController.php(518): call_user_func_array(Array, Array)
  #16 /home/www/analytics/core/FrontController.php(122): Piwik\FrontController->doDispatch('API', false, Array)
  #17 /home/www/analytics/core/dispatch.php(34): Piwik\FrontController->dispatch()
  #18 /home/www/analytics/index.php(27): require_once('/home/www/anal...')
  #19 /home/www/analytics/misc/cron/archive.php(70): require_once('/home/www/anal...')
  #20 {main}
INFO CoreAdminHome[2016-12-26 01:23:00] [5fd6a] Scheduler: finished. Time elapsed: 0.803s
slawa-dev commented 7 years ago

@mattab GeoIP2 Plugin is buggy atm. So the fix might work when the plugin is fixed.

https://github.com/matomo-org/matomo/issues/11157

sgiehl commented 6 years ago

Matomo 3.5.0 now includes a GeoIp2 plugin. This fully support the new mmdb format.