search

satyrius / gonx

Nginx access log parser for Golang
MIT License
351 stars 86 forks source link

Use of regexp.QuoteMeta in NewParser function causes problem with my nginx.conf #31

Open wturyn opened 8 years ago

wturyn commented 8 years ago

My nginx.conf includes following definition of log format (sorry for bad formatting):

log_format log_with_cookie '[$time_local]\t'
         '$remote_addr\t'
        '$geoip_country_code\t'
        '$msec\t'
        '$remote_user\t'
        '"$request"\t'
        '$status\t'
         '$body_bytes_sent\t'
         '$cookie_uid\t'
         '$uid_got\t'
        '$uid_set\t'
        '$cookie_u\t'
        '"$http_accept_language"\t'
         '$cookie_c\t'
         '"$http_user_agent"\t'
         '"$http_referer"\t'
         '"$args"\t'
    '"$sent_http_set_cookie"';

Because of use of regexp.QuoteMeta in NewParser function, tab characters are not properly recognized during parsing.

Is there some particular reason why it is done this way?

satyrius commented 7 years ago

I'm so sorry @wturyn, I dont't know how I missed your message. Is it still actual for you?

wturyn commented 7 years ago

Yes.

satyrius commented 7 years ago

Could you provide a few lines of your log for this format?

wturyn commented 7 years ago 
[07/Jan/2017:22:01:40 +0100]    127.0.0.1   -   1483822900.454  -   "GET /register? HTTP/1.1"   200 875 -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/login"    "-" "-"
[07/Jan/2017:22:01:40 +0100]    127.0.0.1   -   1483822900.470  -   "GET /public/js/jquery-2.2.4.js HTTP/1.1"   404 2696    -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/register?"    "-" "-"
[07/Jan/2017:22:01:40 +0100]    127.0.0.1   -   1483822900.490  -   "GET /public/css/reglog.css HTTP/1.1"   404 2689    -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/register?"    "-" "-"
[07/Jan/2017:22:01:40 +0100]    127.0.0.1   -   1483822900.923  -   "GET /register? HTTP/1.1"   200 875 -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/login"    "-" "-"
[07/Jan/2017:22:01:40 +0100]    127.0.0.1   -   1483822900.938  -   "GET /public/js/jquery-2.2.4.js HTTP/1.1"   404 2696    -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/register?"    "-" "-"
[07/Jan/2017:22:01:40 +0100]    127.0.0.1   -   1483822900.948  -   "GET /public/css/reglog.css HTTP/1.1"   404 2689    -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/register?"    "-" "-"
[07/Jan/2017:22:01:41 +0100]    127.0.0.1   -   1483822901.845  -   "GET /login? HTTP/1.1"  200 815 -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/register?"    "-" "-"
[07/Jan/2017:22:01:41 +0100]    127.0.0.1   -   1483822901.859  -   "GET /public/js/jquery-2.2.4.js HTTP/1.1"   404 2695    -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/login?"   "-" "-"
[07/Jan/2017:22:01:41 +0100]    127.0.0.1   -   1483822901.863  -   "GET /public/css/reglog.css HTTP/1.1"   404 2688    -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/login?"   "-" "-"
[07/Jan/2017:22:01:41 +0100]    127.0.0.1   -   1483822901.886  -   "GET /public/css/reglog.css HTTP/1.1"   404 2688    -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/login?"   "-" "-"
[07/Jan/2017:22:01:47 +0100]    127.0.0.1   -   1483822907.831  -   "GET /register? HTTP/1.1"   200 875 -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/login?"   "-" "-"
[07/Jan/2017:22:01:47 +0100]    127.0.0.1   -   1483822907.847  -   "GET /public/js/jquery-2.2.4.js HTTP/1.1"   404 2696    -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/register?"    "-" "-"
[07/Jan/2017:22:01:47 +0100]    127.0.0.1   -   1483822907.856  -   "GET /public/css/reglog.css HTTP/1.1"   404 2689    -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/register?"    "-" "-"
[07/Jan/2017:22:01:49 +0100]    127.0.0.1   -   1483822909.649  -   "GET / HTTP/1.1"    200 815 -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "-" "-" "-"
[07/Jan/2017:22:01:49 +0100]    127.0.0.1   -   1483822909.664  -   "GET /public/js/jquery-2.2.4.js HTTP/1.1"   404 2689    -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/" "-" "-"
[07/Jan/2017:22:01:49 +0100]    127.0.0.1   -   1483822909.667  -   "GET /public/css/reglog.css HTTP/1.1"   404 2683    -   -   -   -   "pl-PL,pl;q=0.8,en-US;q=0.6,en;q=0.4"   -   "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"  "http://g.com/" "-" "-"