tstack / lnav

Log file navigator
http://lnav.org
BSD 2-Clause "Simplified" License
8.01k stars 321 forks source link

Change Apache log format - how can i change the colors or extend of access.log? #885

Open EmJotGeh opened 3 years ago

EmJotGeh commented 3 years ago

I've made a change to the apache log format.

from: LogFormat "% h% l% u% t \"% r \ "%> s% O \"% {Referer} i \ "\"% {User-Agent} i \ "" combined

to: LogFormat "% {GEOIP_COUNTRY_CODE} e% a% l% u% t \"% r \ "%> s% O \"% {Referer} i \ "\"% {User-Agent} i \ "" combined

This change has an effect on the lnav log format colors and I can't find out, how to change or extend the colors for the access.log for lnav.

I have read the documentation but I can't find the reference to the access.log for lnav.

Would be very grateful for a hint

access_log_Screenshot

tstack commented 3 years ago

It looks like you'll probably need to modify the format a bit or create a new format. What version of lnav are you using?

For reference, the access_log format file is here:

https://github.com/tstack/lnav/blob/master/src/formats/access_log.json

You could try install the following format file fragment which tries to capture the country code:

{
    "access_log": {
        "regex": {
            "with-geoip": {
                "pattern": "^(?<c_country_code>\\w{2}) (?<c_ip>[\\w\\.:\\-]+)\\s+[\\w\\.\\-]+\\s+(?<cs_username>\\S+)\\s+\\[(?<timestamp>[^\\]]+)\\] \"(?:\\-|(?<cs_method>\\w+) (?<cs_uri_stem>[^ \\?]+)(?:\\?(?<cs_uri_query>[^ ]*))? (?<cs_version>[\\w/\\.]+))\" (?<sc_status>\\d+) (?<sc_bytes>\\d+|-)(?: \"(?<cs_referer>[^\"]+)\" \"(?<cs_user_agent>[^\"]+)\")?\\s*(?<body>.*)"
            }
        },
        "value": {
            "c_country_code": {
                "kind": "string",
                "identifier": true
            }
        }
    }
}

You can install it by copying the above snippet into a file named "access_log_with_country_code.json" and copying it into `~/.lnav/formats/installed". You might have to edit hte "with-geoip" pattern to match your log files. If you can provide some anonymized log messages, I can help with that.

(Sorry for the late response)

EmJotGeh commented 3 years ago

(Sorry for the late response)

no problem - thank you for help. I understand the principle but I still have difficulties adapting the format file accordingly and i got a parser error. Maybe you have an idea if I show you an excerpt from the apache access.log file.

209.85.xxx.xxx - - [29/Jul/2021:00:41:32 +0200] "GET /google8191a051d2d54f47.html HTTP/1.1" 200 7176 "-" "Mozilla/5.0 (compatible; Google-Site-Verification/1.0)" 209.85.xxx.xxx - - [29/Jul/2021:00:42:01 +0200] "GET /google444b9ccf685c1cf7.html HTTP/1.1" 200 7176 "-" "Mozilla/5.0 (compatible; Google-Site-Verification/1.0)" 37.138.xxx.xxx - - [29/Jul/2021:01:06:13 +0200] "GET /content/plugins/under-construction-page/themes/css/bootstrap.min.css?v=3.89 HTTP/2.0" 200 20044 "https://my-domain.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0" 37.138.xxx.xxx - - [29/Jul/2021:01:06:13 +0200] "GET /content/plugins/under-construction-page/themes/css/common.css?v=3.89 HTTP/2.0" 200 448 "https://my-domain.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0" 37.138.xxx.xxx - - [29/Jul/2021:01:06:13 +0200] "GET /content/plugins/under-construction-page/themes/max_designer/style.css?v=3.89 HTTP/2.0" 200 752 "https://my-domain.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0" 37.138.xxx.xxx - - [29/Jul/2021:01:06:13 +0200] "GET /content/plugins/under-construction-page/themes/css/font-awesome.min.css?v=3.89 HTTP/2.0" 200 7053 "https://my-domain.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0" 37.138.xxx.xxx - - [29/Jul/2021:01:06:13 +0200] "GET /content/plugins/under-construction-page/themes/max_designer/max_designer.png HTTP/2.0" 200 163011 "https://my-domain.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0" 37.138.xxx.xxx - - [29/Jul/2021:01:06:13 +0200] "GET /content/plugins/under-construction-page/themes/images/favicon.png HTTP/2.0" 200 4026 "https://my-domain.com/" "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0"