Make squidanalyser able to parse multiple access.log concatenation (timecodes not correctly sorted)

darold / squidanalyzer

Squid Analyzer parses Squid proxy access log and reports general statistics about hits, bytes, users, networks, top URLs, and top second level domains. Statistic reports are oriented toward user and bandwidth control.

http://squidanalyzer.darold.net/

125 stars 36 forks source link

Make squidanalyser able to parse multiple access.log concatenation (timecodes not correctly sorted) #182

Closed JFRANCOI closed 5 years ago

JFRANCOI commented 6 years ago

Hello,

I wish to squidanalyze many squid process through a single squidanalyser report. So I've used rsyslog to concatenate access.log from many squid inside a single file that's used by squidanalyzer. The problem is squidanalyzer refuses to process because sometimes timecodes are not correclty sorted between both machines and rsyslog concentrator.

Any advise or new feature ? Regards

lbsou commented 6 years ago

Hi, I had the same problem with multiple squid instances logging to our syslog-ng server.

My fix is not perfect but get the job done

I use a regex to replace the unix time with the current unix time on syslog-ng

subst("^.{10}..{3}", "${UNIXTIME}.000", value("MESSAGE"));

I don't know if this is possible with rsyslog.

Good luck!

darold commented 5 years ago

This feature is now fully implemented in development code, you can give several log files or a single log file without taking care of timestamp ordering. It can also be used with multi-process (with -j option). A new release is coming.