matomo-org / matomo-log-analytics

Import any kind of server logs in Matomo for powerful log analytics. Universal log file parsing and reporting.
https://matomo.org/log-analytics/
GNU General Public License v3.0
225 stars 118 forks source link

rsyslog + nginx live log import silently fail #285

Open 1000i100 opened 4 years ago

1000i100 commented 4 years ago

It probably have some comon point with #258

I've followed the rsyslog part of this : https://github.com/matomo-org/matomo-log-analytics#how-do-i-setup-nginx-to-directly-import-to-matomo-via-syslog

and after some folder manual creation and the change of /var/cache/nginx/access.socket by /dev/log in nginx.conf and 10-matomo.conf if i uncomment the debug line local0.* /var/tmp/nginx.tmp;matomo, log apear in /var/tmp/nginx.tmp

If i try to import them with matomo.sh it show me the script run, but import nothing.

If i try to run import_logs.py itself, it work.

So i suspect there is something not working in /usr/local/matomo/matomo.sh

 cat /var/tmp/nginx.tmp | ./misc/log-analytics/import_logs.py --url=***** --token-auth=******  --enable-http-errors --enable-http-redirects --enable-static --enable-bots --recorders=4 --log-format-name=nginx_json -

found nothing to import but :

./misc/log-analytics/import_logs.py --url=***** --token-auth=******  --enable-http-errors --enable-http-redirects --enable-static --enable-bots --recorders=4 --log-format-name=nginx_json /var/tmp/nginx.tmp

work

Is there some documentation to fix in README.md about rsyslog /usr/local/matomo/matomo.sh syntax :

echo "${@}" | /***MY_MATOMO_FOLDER***/misc/log-analytics/import_logs.py

or is it something to fix in misc/log-analytics/import_logs.py ?

or anything else i've not spotted ?

1000i100 commented 4 years ago

After reading this : https://github.com/matomo-org/matomo-log-analytics/issues/282#issuecomment-716676600 i've updated import_logs.py to the latest from this repository. I've change in first line : #!/usr/bin/python to #!/usr/bin/python3 and it now have the same behavor with this 2 way for calling it :

cat /var/tmp/nginx.tmp | ./misc/log-analytics/import_logs.py [...] -
./misc/log-analytics/import_logs.py [...] /var/tmp/nginx.tmp

But it does'nt import anything :

Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "******/misc/log-analytics/import_logs.py", line 1849, in _run_bulk
    self._record_hits(hits)
  File ""******/misc/log-analytics/import_logs.py", line 1995, in _record_hits
    'requests': [self._get_hit_args(hit) for hit in hits]
  File ""******/misc/log-analytics/import_logs.py", line 1995, in <listcomp>
    'requests': [self._get_hit_args(hit) for hit in hits]
  File ""******/misc/log-analytics/import_logs.py", line 1891, in _get_hit_args
    site_id, main_url = resolver.resolve(hit)
  File ""******/misc/log-analytics/import_logs.py", line 1775, in resolve
    return self._resolve_by_host(hit)
  File ""******/misc/log-analytics/import_logs.py", line 1758, in _resolve_by_host
    site_id = self._resolve(hit)
  File ""******/misc/log-analytics/import_logs.py", line 1726, in _resolve
    res = self._get_site_id_from_hit_host(hit)
  File ""******/misc/log-analytics/import_logs.py", line 1681, in _get_site_id_from_hit_host
    url=hit.host,
  File ""******/misc/log-analytics/import_logs.py", line 1633, in call_api
    return self._call_wrapper(self._call_api, None, None, method, **kwargs)
  File ""******/misc/log-analytics/import_logs.py", line 1585, in _call_wrapper
    response = func(*args, **kwargs)
  File ""******/misc/log-analytics/import_logs.py", line 1574, in _call_api
    return json.loads(res)
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

1365 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)
[... Ctrl+C to stop it ...]
1365 lines parsed, 0 lines recorded, 0 records/sec (avg), 0 records/sec (current)

Logs import summary
-------------------

    0 requests imported successfully
    17 requests were downloads
    0 requests ignored:
        0 HTTP errors
        0 HTTP redirects
        0 invalid log lines
        0 filtered log lines
        0 requests did not match any known site
        0 requests did not match any --hostname
        0 requests done by bots, search engines...
        0 requests to static resources (css, js, images, ico, ttf...)
        0 requests to file downloads did not match any --download-extensions

Website import summary
----------------------

    0 requests imported to 0 sites
        0 sites already existed
        0 sites were created:

    0 distinct hostnames did not match any existing site:

Performance summary
-------------------

    Total time: 149 seconds
    Requests imported per second: 0.0 requests per second

Processing your log data
------------------------

    In order for your logs to be processed by Matomo, you may need to run the following command:
     ./console core:archive --force-all-websites --force-all-periods=315576000 --force-date-last-n=1000 --url='****'