search

matomo-org / matomo-log-analytics

Import any kind of server logs in Matomo for powerful log analytics. Universal log file parsing and reporting.
https://matomo.org/log-analytics/
GNU General Public License v3.0
224 stars 118 forks source link

Don't get any records on Apache #349

Closed Johnnii360 closed 1 year ago

Johnnii360 commented 1 year ago

Hi guys!

I try now for days to get any records for my Whoogle installation. Because tracking code inclusion into the template files failed I want to use now the direct Apache Piped Log way.

But the problem is: I don't get any records.

I followed the example instructions for Apache and set it up like this now:

ErrorLog ${APACHE_LOG_DIR}/whoogle-error.log
CustomLog ${APACHE_LOG_DIR}/whoogle-access.log combined
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" MatomoLogFormat
CustomLog "|/var/www/[CENSORED]/misc/log-analytics/import_logs.py --idsite=[CENSORED] --token-auth=[CENSORED] --enable-http-errors --enable-http-redirects --enable-bots --url=https://[CENSORED] --recorders=4 --log-format-name=common_complete --debug --output=/var/log/apache2/matomo-debug.log -" MatomoLogFormat

I set the right path, the right idsite and authtoken but don't get any records nor errors in my logs. It's like Matomo don't give a fluff about it. I already tried to disable "Do not Track" and used a Web Proxy to check getting any records. Nothing worked!

=== Server Info ===

=== Matomo Info ===

Click to view System Check ### Mandatory checks #### PHP-Version >= 7.2.5: ✔ 8.2.4 #### PDO Erweiterung: ✔ #### PDO\MYSQL Erweiterung: ✔ #### MYSQLI Erweiterung: ✔ #### Weitere erforderliche Anforderungen: ✔ zlib ✔ json ✔ filter ✔ hash ✔ session #### Erforderliche Funktionen: ✔ debug_backtrace ✔ eval ✔ hash ✔ gzcompress ✔ gzuncompress ✔ pack #### Benötigte PHP Konfiguration (php.ini): ✔ session.auto_start = 0 ✔ max_execution_time = 0 OR = -1 OR >= 30 #### Verzeichnisse mit Schreibzugriff: ✔ $DOC_ROOT/tmp ✔ $DOC_ROOT/tmp/assets ✔ $DOC_ROOT/tmp/cache ✔ $DOC_ROOT/tmp/climulti ✔ $DOC_ROOT/tmp/latest ✔ $DOC_ROOT/tmp/logs ✔ $DOC_ROOT/tmp/sessions ✔ $DOC_ROOT/tmp/tcpdf ✔ $DOC_ROOT/tmp/templates_c ### Optional checks #### Verzeichnisse, welche privat sein müssen: ✔ Alle privaten Verzeichnisse sind nicht aus dem Internet zugänglich. #### Empfohlene private Verzeichnisse: ✔ Alle privaten Verzeichnisse sind nicht aus dem Internet zugänglich. #### Dateiintegrität: ✔ #### 64-bit PHP Binär: ✔ #### Tracker-Status: ✔ #### Speicherlimit: ✔ 128M #### Zeitzone: ✔ #### Öffnen einer URL: ✔ curl #### PageSpeed ist deaktiviert: ✔ #### GD > 2.x + FreeType (Grafiken): ✔ #### Andere Erweiterungen: ✔ json ✔ libxml ✔ dom ✔ SimpleXML ✔ openssl #### Andere Funktionen: ✔ shell_exec ✔ set_time_limit ✔ mail ✔ parse_ini_file ✔ glob ✔ gzopen ✔ md5_file #### Dateisystem: ✔ #### Cron einrichten - Prozesse via CLI steuern: ✔ OK #### Letzter erfolgreicher Abschluss der Archivierung: ✔ Der Archivierungsprozess wurde vor 00:04:43 erfolgreich abgeschlossen. #### Datenbankfähigkeiten: ✔ UTF8mb4 charset ✔ LOAD DATA INFILE ✔ CREATE TEMPORARY TABLES ✔ Changing transaction isolation level #### Maximale Packetgröße: ✔ #### Erzwungene SSL Verbindung: ✔ #### Standorterkennung: ✔ geoip2php (continent_code, continent_name, country_code, country_name, region_code, region_name, city_name, postal_code, lat, long) #### Update über HTTPS: ✔ #### Schreibbarer JavaScript-Tracker ("/matomo.js" & "/piwik.js"): ✔ ### Informational results #### Matomo Version: 4.14.1 #### Matomo Update History: 4.11.0,4.10.1,4.8.0,4.6.1,4.5.0,4.4.1 #### Matomo Install Version: Unknown - pre 3.8. #### Latest Available Version: 4.14.1 #### Is Git Deployment: 0 #### PHP_OS: Linux #### PHP SAPI: apache2handler #### Timezone Version: 0.system #### PHP Timezone: UTC #### PHP Time: 1681651484 #### PHP Datetime: 2023-04-16 13:24:44 #### PHP INI max_execution_time: 30 #### PHP INI post_max_size: 8M #### PHP INI max_input_vars: 1000 #### PHP INI zlib.output_compression: #### Curl Version: 7.74.0, OpenSSL/1.1.1n #### Suhosin Installed: 0 #### DB Prefix: piwik_ #### DB Charset: utf8mb4 #### DB Adapter: PDO\MYSQL #### MySQL Version: 10.5.18-MariaDB-0+deb11u1 #### Num Tables: 525 #### Browser Segment Archiving Enabled: 1 #### Development Mode Enabled: 0 #### Internet Enabled: 1 #### Multi Server Environment: 0 #### Auto Update Enabled: 1 #### Custom User Path: 0 #### Custom Include Path: 0 #### Release Channel: latest_stable #### Plugins Activated: API, Actions, Annotations, BulkTracking, Contents, CoreAdminHome, CoreConsole, CoreHome, CorePluginsAdmin, CoreUpdater, CoreVisualizations, CoreVue, CustomDimensions, CustomJsTracker, CustomVariables 4.1.3, DBStats, Dashboard, DevicePlugins, DevicesDetection, Diagnostics, Events, GeoIp2, Heartbeat, ImageGraph, Insights, Installation, Intl, IntranetMeasurable, LanguagesManager, Login, Marketplace, MobileAppMeasurable, Monolog, Morpheus, MultiSites, Overlay, PagePerformance, PrivacyManager, Provider 4.0.5, Proxy, Referrers, Resolution, SEO, ScheduledReports, SegmentEditor, SitesManager, Tour, Transitions, TwoFactorAuth, UserCountry, UserCountryMap, UserId, UserLanguage, UsersManager, VisitFrequency, VisitTime, VisitorInterest, VisitsSummary, WebsiteMeasurable, Widgetize #### Plugins Deactivated: Ecommerce, ExampleRssWidget 1.0, Feedback, Goals, Live, MobileMessaging, ProfessionalServices, RssWidget, TagManager, Zeitgeist 4.14.1 #### Plugins Invalid: #### Server Info: Apache #### Had visits in last 1 day: 1 #### Had visits in last 3 days: 1 #### Had visits in last 5 days: 1 #### Archive Time Last Started: 2023-04-16 13:20:01 #### Archive Time Last Finished: 2023-04-16 13:20:01 #### User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 #### Browser Language: de-de,de,en-us,en #### Total Invalidation Count: 0 #### In Progress Invalidation Count: 0 #### Scheduled Invalidation Count: 0 #### Earliest invalidation ts_started: #### Latest invalidation ts_started: #### Earliest invalidation ts_invalidated: #### Latest invalidation ts_invalidated: #### Number of segment invalidations: 0 #### Number of plugin invalidations: 0 #### List of plugins being invalidated: #### Anonymize Referrer: #### Do Not Track enabled: 1

If I forgot some infos please tell me.

Edit 2023-04-16, 10:09am: Okay I found now out that the piped logging doesn't work. I additionally created an testlog an imported the data from it manually via bash shell and it worked. Any ideas?

Edit 2023-04-16, 3:06pm: Added Matomo syscheck and fiddled a bit around. It's very weird. When I remove %v from LogFormat and reload Apache it imports "pending" records. When I try the use a Online Proxy to simulate an visitor there is nothing anymore. The debug-log look every time like this:

2023-04-16 15:16:38,512: [DEBUG] Accepted hostnames: whoogle.[CENSORED].de
2023-04-16 15:16:38,512: [DEBUG] Accepted hostnames: piped.[CENSORED].de,pipedapi.[CENSORED].de
2023-04-16 15:16:38,512: [DEBUG] Matomo Tracker API URL is: https://analytics.[CENSORED].de
2023-04-16 15:16:38,512: [DEBUG] Matomo Tracker API URL is: https://analytics.[CENSORED].de
2023-04-16 15:16:38,512: [DEBUG] Matomo Analytics API URL is: https://analytics.[CENSORED].de
2023-04-16 15:16:38,512: [DEBUG] Matomo Analytics API URL is: https://analytics.[CENSORED].de
2023-04-16 15:16:38,512: [DEBUG] Authentication token token_auth is: [CENSORED]
2023-04-16 15:16:38,512: [DEBUG] Authentication token token_auth is: [CENSORED]
2023-04-16 15:16:38,512: [DEBUG] Resolver: static
2023-04-16 15:16:38,512: [DEBUG] Resolver: static
2023-04-16 15:16:38,666: [DEBUG] Launched recorder
2023-04-16 15:16:38,666: [DEBUG] Launched recorder
2023-04-16 15:16:38,666: [DEBUG] Launched recorder
2023-04-16 15:16:38,666: [DEBUG] Launched recorder

So for my opinion it's not my server config rather than your Python script.

Johnnii360 commented 1 year ago

Moved issue to forum due to user/developer inactivity here.