matomo-org / matomo-log-analytics

Import any kind of server logs in Matomo for powerful log analytics. Universal log file parsing and reporting.
https://matomo.org/log-analytics/
GNU General Public License v3.0
224 stars 118 forks source link

Problem importing logs to cPanel install #348

Closed Tom-Jojo closed 1 year ago

Tom-Jojo commented 1 year ago

Hello,

New to Matomo since GA is more or less illegal in EU now.

Installed latest version via Softaculous and tried to import old log files. Unable to, got error like below. Spun up a new Ubuntu and manually installed. Imported all logs without issue (so formatting should be OK. They are all standard cPanel logs). Dumped DB and imported in cPanel install. Everything working fine. I since noticed I misses a few log files, and tried to import in the production system, but get the following errors, both importing locally on cPanel and externally from Ubuntu.

Any ideas ? /Tom

((matomo.example.com:3.7)) [user@2000 log-analytics]$ python import_logs.py --url=https://matomo.example.com --idsite=2 *.gz
Traceback (most recent call last):
  File "import_logs.py", line 1588, in _call_api
    return json.loads(res)
  File "/opt/alt/python37/lib64/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/opt/alt/python37/lib64/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/opt/alt/python37/lib64/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "import_logs.py", line 2688, in <module>
    resolver = config.get_resolver()
  File "import_logs.py", line 1131, in get_resolver
    return StaticResolver(self.options.site_id)
  File "import_logs.py", line 1664, in __init__
    'SitesManager.getSiteFromId', idSite=self.site_id
  File "import_logs.py", line 1647, in call_api
    return self._call_wrapper(self._call_api, None, None, method, **kwargs)
  File "import_logs.py", line 1599, in _call_wrapper
    response = func(*args, **kwargs)
  File "import_logs.py", line 1590, in _call_api
    raise urllib.error.URLError('Matomo returned an invalid response: ' + res.decode("utf-8") )
AttributeError: 'str' object has no attribute 'decode'
((matomo.example.com:3.7)) [user@2000 log-analytics]$

The installation seems OK:

Click to view System Check ### Mandatory checks #### PHP version >= 7.2.5: ✔ 8.1.16 #### PDO extension: ✔ #### PDO\MYSQL extension: ✔ #### MYSQLI extension: ✔ #### Other required extensions: ✔ zlib ✔ json ✔ filter ✔ hash ✔ session #### Required functions: ✔ debug_backtrace ✔ eval ✔ hash ✔ gzcompress ✔ gzuncompress ✔ pack #### Required PHP configuration (php.ini): ✔ session.auto_start = 0 ✔ max_execution_time = 0 OR = -1 OR >= 30 #### Directories with write access: ✔ $DOC_ROOT/tmp ✔ $DOC_ROOT/tmp/assets ✔ $DOC_ROOT/tmp/cache ✔ $DOC_ROOT/tmp/climulti ✔ $DOC_ROOT/tmp/latest ✔ $DOC_ROOT/tmp/logs ✔ $DOC_ROOT/tmp/sessions ✔ $DOC_ROOT/tmp/tcpdf ✔ $DOC_ROOT/tmp/templates_c ### Optional checks #### Required Private Directories: ✔ All private directories are inaccessible from the internet. #### Recommended Private Directories: ✔ All private directories are inaccessible from the internet. #### File integrity: ⚠ Warning: File integrity check failed and reported some errors. You should fix this issue and then refresh this page until it shows no error.

Files were found in your Matomo, but we didn't expect them.
--> Please delete these files to prevent errors. #### 64-bit PHP Binary: ✔ #### Tracker status: ✔ #### Memory limit: ✔ 512M #### Time zone: ✔ #### Open URL: ✔ curl #### PageSpeed is turned off: ✔ #### GD > 2.x + FreeType (graphics): ✔ #### Other extensions: ✔ json ✔ libxml ✔ dom ✔ SimpleXML ✔ openssl #### Other functions: ✔ shell_exec ✔ set_time_limit ✔ mail ✔ parse_ini_file ✔ glob ✔ gzopen ✔ md5_file #### Filesystem: ✔ #### Set up Cron - Managing processes via CLI: ✔ Ok #### Last Successful Archiving Completion: ✔ The archiving process completed successfully 00:19:36 ago. #### Database abilities: ✔ UTF8mb4 charset ⚠ Warning: LOAD DATA INFILE
Hint: Using LOAD DATA INFILE by updating your PHP and MySQL software and ensuring your database user has the FILE privilege speeds up Matomo's archiving process a lot.
You should fix this problem if your Matomo server tracks high-traffic websites (e.g. over 100,000 pages per month). ✔ CREATE TEMPORARY TABLES ✔ Changing transaction isolation level #### Max Packet Size: ✔ #### Forced SSL Connection: ✔ #### Geolocation: ✔ geoip2php (continent_code, continent_name, country_code, country_name, region_code, region_name, city_name, postal_code, lat, long) #### Update over HTTPS: ✔ #### Writable JavaScript Tracker ("/matomo.js"): ✔ ### Informational results #### Matomo Version: 4.13.3 #### Matomo Update History: 4.13.3, #### Matomo Install Version: 4.13.3 #### Latest Available Version: 4.13.3 #### Is Git Deployment: 0 #### PHP_OS: Linux #### PHP_BINARY: /usr/local/bin/lsphp #### PHP SAPI: litespeed #### Timezone Version: 0.system #### PHP Timezone: UTC #### PHP Time: 1678965891 #### PHP Datetime: 2023-03-16 11:24:51 #### PHP INI max_execution_time: 90 #### PHP INI post_max_size: 512M #### PHP INI max_input_vars: 1000 #### PHP INI zlib.output_compression: 1 #### Curl Version: 7.88.0, OpenSSL/1.1.1t #### Suhosin Installed: 0 #### DB Prefix: matomo_ #### DB Charset: utf8mb4 #### DB Adapter: MYSQLI #### MySQL Version: 10.3.38-MariaDB-cll-lve #### Num Tables: 191 #### Browser Segment Archiving Enabled: 1 #### Development Mode Enabled: 0 #### Internet Enabled: 1 #### Multi Server Environment: 0 #### Auto Update Enabled: 1 #### Custom User Path: 0 #### Custom Include Path: 0 #### Release Channel: latest_stable #### Plugins Activated: API, Actions, Annotations, BotTracker 2.07, BraveDetector 0.1.2, BulkTracking, Contents, CoreAdminHome, CoreConsole, CoreHome, CorePluginsAdmin, CoreUpdater, CoreVisualizations, CoreVue, CustomDimensions, CustomJsTracker, Dashboard, DevicePlugins, DevicesDetection, Diagnostics, Ecommerce, Events, Feedback, GeoIp2, Goals, Heartbeat, IP2Location 4.1.2, IP2Proxy 0.1.3, ImageGraph, Insights, Installation, Intl, IntranetMeasurable, InvalidateReports 4.1.1, LanguagesManager, Live, Login, Marketplace, MobileMessaging, Monolog, Morpheus, MultiSites, Overlay, PagePerformance, PrivacyManager, ProfessionalServices, Provider 4.0.5, Proxy, Referrers, Resolution, RssWidget, SEO, ScheduledReports, SegmentEditor, SitesManager, Tour, TrackingSpamPrevention 4.1.7, Transitions, TreemapVisualization 4.0.2, TwoFactorAuth, UserCountry, UserCountryMap, UserId, UserLanguage, UsersManager, VisitFrequency, VisitTime, VisitorInterest, VisitsSummary, WebsiteMeasurable, Widgetize #### Plugins Deactivated: DBStats, MobileAppMeasurable, TagManager #### Plugins Invalid: #### Server Info: LiteSpeed #### Had visits in last 1 day: 1 #### Had visits in last 3 days: 1 #### Had visits in last 5 days: 1 #### Archive Time Last Started: 2023-03-16 11:05:04 #### Archive Time Last Finished: 2023-03-16 11:05:15 #### User Agent: Mozilla/5.0 (Windows NT 10.0; rv:111.0) Gecko/20100101 Firefox/111.0 #### Browser Language: en-us,en #### Total Invalidation Count: 0 #### In Progress Invalidation Count: 0 #### Scheduled Invalidation Count: 0 #### Earliest invalidation ts_started: #### Latest invalidation ts_started: #### Earliest invalidation ts_invalidated: #### Latest invalidation ts_invalidated: #### Number of segment invalidations: 0 #### Number of plugin invalidations: 0 #### List of plugins being invalidated: #### Anonymize Referrer: #### Do Not Track enabled: 0
Tom-Jojo commented 1 year ago

Messing around with it for a while, and it started working then the Python App virtual environment "Application root" was changed from /home/user/matomo.example.com/ to /home/user/matomo.example.com/misc/log-analytics