search

matomo-org / matomo-log-analytics

Import any kind of server logs in Matomo for powerful log analytics. Universal log file parsing and reporting.
https://matomo.org/log-analytics/
GNU General Public License v3.0
224 stars 118 forks source link

import_logs breaks on _resolve_by_host #298

Open rivoduck opened 3 years ago

rivoduck commented 3 years ago

After upgrading to Matomo 4.1.0 import_logs stopped working

If I use the same commandline with python2 and the old version of the script it works fine against the same server (the server I just upgraded to Matomo 4.1.0).

Here are the details! Thank you, Andrea

python3 import_logs.py --url http://myserver --log-hostname my_log_hostname --add-sites-new-hosts --enable-http-errors --enable-http-redirects --enable-static --enable-bots --strip-query-string --recorders 1 --token-auth mytoken mynginxlogfile.log

Parsing log mynginxlogfile.log... Exception in thread Thread-2: Traceback (most recent call last): File "import_logs.py", line 1752, in _resolve_by_host site_id = self._cache[hit.host] KeyError: 'my_log_hostname'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "/usr/lib/python3.5/threading.py", line 862, in run self._target(self._args, self._kwargs) File "/home/topix/matomo_importlogs/import_logs.py", line 1848, in _run_bulk self._record_hits(hits) File "/home/topix/matomo_importlogs/import_logs.py", line 1994, in _record_hits 'requests': [self._get_hit_args(hit) for hit in hits] File "/home/topix/matomo_importlogs/import_logs.py", line 1994, in 'requests': [self._get_hit_args(hit) for hit in hits] File "/home/topix/matomo_importlogs/import_logs.py", line 1890, in _get_hit_args site_id, main_url = resolver.resolve(hit) File "/home/topix/matomo_importlogs/import_logs.py", line 1774, in resolve return self._resolve_by_host(hit) File "/home/topix/matomo_importlogs/import_logs.py", line 1757, in _resolve_by_host site_id = self._resolve(hit) File "/home/topix/matomo_importlogs/import_logs.py", line 1725, in _resolve res = self._get_site_id_from_hit_host(hit) File "/home/topix/matomo_importlogs/import_logs.py", line 1680, in _get_site_id_from_hit_host url=hit.host, File "/home/topix/matomo_importlogs/import_logs.py", line 1632, in call_api return self._call_wrapper(self._call_api, None, None, method, kwargs) File "/home/topix/matomo_importlogs/import_logs.py", line 1584, in _call_wrapper response = func(args, **kwargs) File "/home/topix/matomo_importlogs/import_logs.py", line 1573, in _call_api return json.loads(res) File "/usr/lib/python3.5/json/init.py", line 312, in loads s.class.name)) TypeError: the JSON object must be str, not 'bytes'

kiwipok commented 3 years ago

Hi,

I have this identical problem with matomo-log-analytics. After moving from 3.5.1 to Matomo 4.2.1 import_logs stopped working: ..... Exception in thread Thread-1: Traceback (most recent call last): File "/srv/piwik/misc/log-analytics/import_logs.py", line 1757, in _resolve_by_host site_id = self._cache[hit.host] KeyError: 'xxx.yyy.zzz'

During handling of the above exception, another exception occurred: .....

Is there a solution to this issue so impacting?