search

allinurl / goaccess

GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.
https://goaccess.io
MIT License
18.18k stars 1.1k forks source link

Support for FastMail CSV access log format? #1032

Closed mignon-p closed 6 years ago

mignon-p commented 6 years ago

My apologies if this is obvious, but I don't know that much about website logs.

I'm using FastMail to serve files. FastMail has the option to download access logs in the form of a CSV file. The file looks like this:

Stamp,HostName,Page,IpAddress,Referer,UserAgent
"2018-02-13 13:22:46",funwithsoftware.org,/atom.xml,146.148.67.171,,Java/1.8.0_121
"2018-02-13 13:27:22",funwithsoftware.org,/images/2016-tinkercad.png,54.36.148.34,,"Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"
"2018-02-13 13:28:32",funwithsoftware.org,/atom.xml,146.148.67.171,,Java/1.8.0_121
"2018-02-13 13:34:18",funwithsoftware.org,/atom.xml,146.148.67.171,,Java/1.8.0_121
"2018-02-13 13:39:53",funwithsoftware.org,/atom.xml,146.148.67.171,,Java/1.8.0_121
"2018-02-13 13:46:27",funwithsoftware.org,/atom.xml,146.148.67.171,,Java/1.8.0_121
"2018-02-13 13:48:12",funwithsoftware.org,/images/2017-Haskell-LCD-800px.jpg,192.30.252.47,,"github-camo (65c9ebe0)"
"2018-02-13 13:50:52",funwithsoftware.org,/posts/2014-05-23-crying-wolf.html,216.244.66.247,,"Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)"
"2018-02-13 13:50:52",funwithsoftware.org,/posts/2015-03-27-no-firmware-upgrade-goes-unpunished.html,216.244.66.247,,"Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)"
"2018-02-13 13:52:33",funwithsoftware.org,/,54.235.20.221,,"CCBot/2.0 (http://commoncrawl.org/faq/)"

Is there a way to set up GoAccess to parse this format? Thanks!

allinurl commented 6 years ago

Give this a shot:

goaccess access.log --log-format='"%d %t",%v,%U,%h,%^,"%u"' --date-format=%Y-%m-%d --time-format=%T --http-method=no --http-protocol=no --ignore-panel=REFERRING_SITES --ignore-panel=STATUS_CODES
mignon-p commented 6 years ago

Thanks! That worked really well. However, I changed the %^ to %R to support the referrer:

goaccess accesslog-2.csv --log-format='"%d %t",%v,%U,%h,%R,"%u"' --date-format=%Y-%m-%d --time-format=%T --http-method=no --http-protocol=no --ignore-panel=STATUS_CODES --ignore-panel=NOT_FOUND -o accesslog-2.html