common log format error

nkravi commented 8 years ago

burger.letters.com - - [01/Jul/1995:00:00:12 -0400] "GET /shuttle/countdown/video/livevideo.gif HTTP/1.0" 200 0

the ip of type burger.letters.com is not getting parsed correctly

megastef commented 8 years ago

Which Docker Container do you use?

megastef commented 8 years ago

In any case here is pattern you need:


- #  Web Logs
  sourceName: !!js/regexp /httpd|access_log|apache2|nginx/
  match:
    - regex: !!js/regexp /^(\S+)\s+(-|.+?)\s+(-|.+?)\s+\[(.*)\]\s\"(\S+)\s(\S+)\s(\S+)\s(\d+)\s(\d+)/i
      type: access_common
      fields:       [client_ip,remote_id,user,ts,method,path,http_version,status_code,size]
      dateFormat: DD/MMM/YYYY:HH:mm:ss ZZ
      transform: !!js/function >
        function (p) {
          p.message = p.method + ' ' + p.path
        }

megastef commented 8 years ago

The pattern is now in https://github.com/sematext/logagent-js/commit/9d9b5d225765e800e38ce61ceebd830946531be3 and will be in the next npm version 1.1.18 after some tests with NGINX, Apache HTTP and other web servers. But you can download https://raw.githubusercontent.com/sematext/logagent-js/master/patterns.yml and use this file via command line

wget https://raw.githubusercontent.com/sematext/logagent-js/master/patterns.yml -o mypattern.yml
tail -f /var/log/access_log | logagent  -f ./mypattern.yml -y -n httpd

Or use in sematext/sematext-agent-docker

docker run ... -v $PWD/mypattern.yml:/etc/logagent/patterns.yml

sematext / logagent-js

common log format error #3