no TOP Sites, etc. on Month Report

[eviltrooper]

Hello, i used the version 6.6 but i missing my montly TOP URLS TOP Domains. the programm dont make the user.html, url.html etc. also the "dat* files are missing.

on the weekly and day statistics all is working, only mothly doesnt work.

i run squidanaylser with -j 4 and with --no-year-stat

[darold]

Yes, that the normal behavior, with --no-year-stat enabled we only store global statistics in year and month views, users and urls stats are not reported outside daily and weekly reports. This is for performances reason.

[eviltrooper]

okay. so i need to run it without "--no-year-stat" ? because on older Versions, did it work

[darold]

Yes, if performances are acceptable for you.

[eviltrooper]

it must, because we uses the Monthy Top statistics.

[eviltrooper]

mhh with "d -j 4 -b 2017-05 " to rebuild month "May it still doesn' work. `Not Found

The requested URL /squid-reports/proxy03/2017/05/url.html was not found on this server.`

[darold]

This is because there is no .dat files. You have to parse your log again, the rebuild action just recreate report from the data files, but when using --no-year-stat no data file are generated. What is the size of your log file ?

[eviltrooper]

ahh okay.. mhhh i delete my Month Logfile every Month :( so i need to get my Backup file and recreate it. i give it a try.

Could you build it back so that he make an Month report without a Year statistic?

[darold]

No not in current code, but I can add this feature.

[eviltrooper]

it has 1,2GB but it is only for "May"

so i need to do this "/usr/bin/squid-analyzer -c /etc/squidanalyzer/squidanalyzer.proxy03.conf.tmp -d -j 4 -r -b 2017-05" ?

in this conf file the Logfile shows to my "May" logfile. is this correct?

[darold]

You first need to remove the 2017/05 and 2017/06 directories then run squid-analyzer as follow:

/usr/bin/squid-analyzer -c /etc/squidanalyzer/squidanalyzer.proxy03.conf.tmp -d -j 4 access-2017-05.log access-2017-06.log

You can give the log file as arguments of the command line, it will overridden the configuration setting of LogFile.

[eviltrooper]

hi, can you add the feature again to do month rerports without a Year statistic? that would be great, or i must go back to an old version :(

[darold]

Hi,

Yes sorry for the delay but it is very hard for me to find spare time these days, I will try to work on it this weekend. If you see that this is not done in next Wednesday please ping me again.

Regards,

[eviltrooper]

that would be very cool!! if you need more time no Problem :)

the strange think is is it normal that he needs over 2 Days for the year statistics with 4 processors?

[darold]

Perhaps this could be improved but I don't have such huge data directory, do you think it is possible to send me to my private email a link to download the data directory and your /etc/squidanalyzer/ configuration files?

[eviltrooper]

mhh could you need the data directory or the log files.. the problem is the data directroy is very big. 12gigabyte

[darold]

The log files would be enough.

[eviltrooper]

mhh i only have log files from the last 3 months. but they are from our Company, it would not be so good to share this i think.

[darold]

I'm the only one who will have access to your log if you send the link to my private email but I understand that this can be a problem for your company. That's not a problem I will try to deduce what is making report generation so slow.

[eviltrooper]

if the month Top statistics will work witouht year statistic again i would be happy :) i dont need the years statistics

[darold]

Commit 58bdeda adds the --with-month-stat command line to force compute month statistics when --no-year-stat is used.

[darold]

Please give it a try and let me know if that's fit your needs.

[eviltrooper]

root@sipka043l:/var/tmp# squid-analyzer -c /etc/squidanalyzer/squidanalyzer.proxy03.conf.tmp -b 2017-04 -d -j 5 --no-year-stat --with-month-stat proxy03april.log

SquidAnalyzer version 6.6 Building HTML output into /var/www/squid-reports/proxy03/ Generating statistics for year 2017 User statistics in /var/www/squid-reports/proxy03//2017... Top URL statistics in /var/www/squid-reports/proxy03//2017... Mime type statistics in /var/www/squid-reports/proxy03//2017... Top denied URL statistics in /var/www/squid-reports/proxy03//2017... Network statistics in /var/www/squid-reports/proxy03//2017... Top domain statistics in /var/www/squid-reports/proxy03//2017... Cache statistics in /var/www/squid-reports/proxy03//2017...

mhh.. still running now for 11 Hours.. normal???

Logfile is 1.1gb

root@sipka043l:/var/tmp# ls -larh proxy03april.log -rw-r--r-- 1 root root 1,1G Jul 13 12:56 proxy03april.log

[darold]

Hi,

It takes around 20 minutes for me on a 1.4GB file using the same command but with -j4 and the file only cover 7 days. Perhaps you have some network or user aliases in your configuration file that makes things slower? If you want me to help more, you can send to my private email a link to download your gzipped log file and your configuration files.

[eviltrooper]

i send you an email!

[eviltrooper]

Hello, anything new about my issue? i sended you an gzipped log file... i still have issue with the Month statistics.

[darold]

I'm closing this issue because I think we have reach the limit of the actual SquidAnalyzer design. Better performances will be possible but it need a major rewrite of SA.