search

darold / squidanalyzer

Squid Analyzer parses Squid proxy access log and reports general statistics about hits, bytes, users, networks, top URLs, and top second level domains. Statistic reports are oriented toward user and bandwidth control.
http://squidanalyzer.darold.net/
125 stars 36 forks source link

'Denied' urls on the 'User Statistics' page #75

Closed tierpod closed 8 years ago

tierpod commented 9 years ago

Hello! Thank you for useful project!

It there any way to show 'Denied' urls on the 'User Statistics' page?

darold commented 9 years ago

Latest commits add this feature through a new Top Denied menu link.

Regards,

tierpod commented 9 years ago

Thank you, darold! It's work like a charm! The best squid reporting tool.

mrv84 commented 9 years ago

Hi darold, and thank you for the great work. I got a problem, i just installed squid proxy analyzer on centos 7 everythings works fine but i can't see on my report the "top denied" menu

can you help me?

darold commented 9 years ago

Hi,

It should just works, do you have UrlReport enabled ? If yes please use latest code from github and let me know if you still have the same issue.

Best regards,

mrv84 commented 9 years ago

Hi darld and thanks for your fast reply Yes Urlreport is enabled. Sorry, i'm not familiar with github, what kind of code i have to use? Thank you so much

Regards,

darold commented 9 years ago

On the main github page of the project you have a "Dowload ZIP" button on the right part or you can use the following command:

wget  https://github.com/darold/squidanalyzer/archive/master.zip

Or if you want to use git, just perform the following command:

git clone git@github.com:darold/squidanalyzer.git

this is different way to download latest code.

Regards,

mrv84 commented 9 years ago

Hi darold and thanks for your support now, i can see the new "top denied" menu

I'm using squidguard , but i can't see the url blocked by blacklist config Only the url denied on squid acl

I made some missconfiguration? thanks for your support

darold commented 9 years ago

Please post here a part of you log file with denied Url.

mrv84 commented 9 years ago

Hi darold this is what /var/log/squid/access.log shows me for URL "blacklisted" with squidguard (for example facebook)

1434702449.487 62 10.0.21.76 TCP_MISS/301 658 GET https://www.facebook.com/ - HIER_DIRECT/10.0.21.35 text/html

and that's the result of a specified denied acl on squid.con

1434708762.522 97 10.0.21.76 TCP_DENIED/403 3752 GET http://html.it/ - HIER_NONE/- text/html

On squidanalyer i can see the blocked URL by squid , not the "filtered" url by squidguard

darold commented 9 years ago

Ok, that's normal. If you have a TCP_MISS instead of a TCP_DENIED SquidAnalyzer has no way to know that SquidGuard has blacklisted the url.

mrv84 commented 9 years ago

Darold you have been very kind in your support, now i'll make probably a question out of context . Did you know if there is a way to tell squidguard to generate a tcp_denied instead of a tcp_miss ??

darold commented 9 years ago

Hi,

This is not SquidGuard that generates this TCP_MISS, this is Squid. The response code from SquidGuard in case of rules violation seems to be 301, but the main problem is to be able to identify that this http response code comes from SquidGuard and not an other site.

mrv84 commented 9 years ago

So actually there's no way to use proxyanalyzer in combination with "squidguard" and no way to get your great software populated with the data of the "filtered" sites? It's something that it's completly unresolvable or you are working on that?

darold commented 9 years ago

Do you have a log from squidguard that you can send me? This will help me to see if it can be included in the SquidAnalyzer report.

mrv84 commented 9 years ago

Hi Darold This is the example log generated by squidguard for two different blacklist (dest on squidGuard.conf)

First (log saved in /var/log/squidGuard/blkshop.log) 2015-06-23 15:32:15 [15717] Request(mydesk/shopping/-) http://www.amazon.it/ 10.0.21.76/- - GET REDIRECT 2015-06-23 15:35:56 [15717] Request(mydesk/shopping/-) http://www.ebay.it/ 10.0.21.76/- - GET REDIRECT

Second (log saved in /var/log/squidGuard/blkwebmail.log) 2015-06-23 15:34:01 [15717] Request(mydesk/webmail/-) http://www.hotmail.it/ 10.0.21.76/- - GET REDIRECT 2015-06-23 15:41:59 [15717] Request(mydesk/webmail/-) http://hotmail.com.tr/ 10.0.21.76/- - GET REDIRECT

This is a log (source on squidGuard.conf) generated by source

2015-06-23 15:48:17 [16000] Request(mydesk/news/-) http://geoisp.virgilio.it/ioladv/iolobj-rc-read.js?tm=1435067304772 10.0.21.76/- - GET REDIRECT 2015-06-23 15:48:36 [16000] Request(mydesk/news/-) http://adimg.virgilio.it/tracks/bi/images/bi_clk.gif?pmk=optin_viewtoolbar&rand=7488606 10.0.21.76/- - GET REDIRECT 2015-06-23 15:52:56 [16000] Request(mydesk/social/-) https://www.facebook.com/ 10.0.21.76/- - GET REDIRECT 2015-06-23 15:55:54 [16000] Request(mydesk/social/-) http://www.linkedin.com/ 10.0.21.76/- - GET REDIRECT 2015-06-23 15:56:34 [16000] Request(mydesk/news/-) http://www.corriere.it/ 10.0.21.76/- - GET REDIRECT 2015-06-23 15:58:17 [16000] Request(mydesk/gamble/-) http://www.pokerstars.it/ 10.0.21.76/- - GET REDIRECT

mrv84 commented 9 years ago

Hi Darold, any news?

darold commented 9 years ago

I'm waiting for a contributor reply on pull request #88 to see if that code can be a solution for you. Otherwise my though was to allow SquidAnalyzer to parse a squidguard log after the access log to retrieve redirection from the log file and add them into the DENIED statistics.

mrv84 commented 9 years ago

Hi Darold how are you? new istructions for me?

darold commented 9 years ago

Hi,

Sorry but you have to wait that I found time to develop the feature, the pull request above only concern squid log entries with TCP_REDIRECT/302. But I see that you are logging to different files, is it possible for you to log in a single file?

mrv84 commented 9 years ago

What i provide to you were some different log generated by SQUIDGUARD based on blacklist defined. For that reason there are two files. Is not a problem to log everything on a single file.

The pull request shows the squid access.log placed on . I just posted the squid log on past , what i can see is a TCP_MISS/301

darold commented 9 years ago

Hi,

Last development code adds support to squidguard log file. You simply have to add the squidguard log file to the list of log files that must be parsed, either in the LogFile configuration directive log list, either at command line, for example:

squid-analyzer /var/log/squid3/access.log /var/log/squid/SquidGuard.log

SquidAnalyzer will automatically detect the log format and report SquidGuard ACL's redirection to the Denied Urls report.

Let me know.

Regards,

mrv84 commented 8 years ago

Hi Darold I just modified the configuration file adding Set the path to the Squid log file LogFile /var/log/squid/access.log LogFile /var/log/squidGuard/squid-bl.log

(i just pointed all the bl log into a single file log) The file were corretly wrote, but i can't see any update on TOP DENIED menu

What's wrong with my configuration?

darold commented 8 years ago

Set it as follow:

LogFile /var/log/squid/access.log,/var/log/squidGuard/squid-bl.log

or give the files at command line.

Update: use coma as separator list.

mrv84 commented 8 years ago

Hi darold

modified the squidanalyzer.conf as you indicated

Set the path to the Squid log file LogFile /var/log/squid/access.log,/var/log/squidGuard/squid-bl.log

the squid-bl.log were correctly populated [root@localhost ~]# tail -f /var/log/squidGuard/squid-bl.log 2015-08-05 18:53:24 [2119] Request(desk-my/social/-) http://www.linkedin.com/ 10.0.21.76/- - GET REDIRECT 2015-08-05 18:54:31 [2119] Request(desk-my/sports/-) http://www.sportmediaset.mediaset.it/ 10.0.21.76/- - GET REDIRECT 2015-08-06 15:26:51 [2107] Request(desk-my/social/-) https://www.facebook.com/ 10.0.21.76/- - GET REDIRECT 2015-08-06 15:27:52 [2107] Request(desk-my/social/-) http://www.twitter.it/ 10.0.21.76/- - GET REDIRECT 2015-08-06 17:17:26 [2107] Request(desk-my/social/-) https://www.facebook.com/ 10.0.21.76/- - GET REDIRECT 2015-08-06 17:17:37 [2107] Request(desk-my/sports/-) http://www.sportmediaset.mediaset.it/ 10.0.21.76/- - GET REDIRECT

but i can't see this information on the top denied menu

darold commented 8 years ago

Hi,

Here is what I can see when using your log entries.

sa-sg

Could you please verify that you still have the issue if you moved the SquidAnalyzer.current file from the output directory and give only the squidguard log file in your squidanalyzer.conf file? Don't forget to restore the SquidAnalyzer.current file after this try.

mrv84 commented 8 years ago

Hi darold i just tried as you indicated and now the topdenied menu shows correctly the information of squid-bl.log

So, what i have to do to make both to correctly work?

darold commented 8 years ago

Ok, I think the issue comes from the timestamp in the log that do not correspond to the last time registered by squidanalyzer. I will check what's going wrong.

lsaluso commented 8 years ago

Hi darold, i have same problem than mrv84, and when i delete SquidAnalyzer.current, top denied option show correctly information. Do you need logs files? I can send you some of the logs to analyze it.

A feature that would be great SquidAnalyzer would have the possibility of showing ("strikethrough" or "highlighted") urls visited and denied by squidguard, in the "users" report.

And Thank you very much for such a useful tool.

darold commented 8 years ago

Please update to latest development code, there is two fix in last commit 928ed13 that might fix your issue.

mrv84 commented 8 years ago

Hi Darold Can you explain the right procedure for a correct and fully functionally update? Many thanks

tierpod commented 8 years ago

@mrv84 Just download last development source code from git:

git clone https://github.com/darold/squidanalyzer.git
# or
git clone git@github.com:darold/squidanalyzer.git

And install this version exactly the same as the last stable version.

mrv84 commented 8 years ago

i need to remove something from previous installation?

darold commented 8 years ago

+1, SquidAnalyzer will take care to not override your existing configuration files. All resources files (js and css) will be overriden. To resume:

perl Makefile.PL
make
sudo make install

Regards,

darold commented 8 years ago

No, you don't need to touch anything. On certain condition it is required to execute squid-analyzer with the --rebuild information but it is always indicated in the ChangeLog and the release note. Here it is not required.

tierpod commented 8 years ago

@darold I think, you can close this issue. Thanks for all! :+1:

mrv84 commented 8 years ago

I just completed the instructions. everything seems to be ok, but the menù "users" now is empty

darold commented 8 years ago

Does your squidanalyzer.conf have been wrongly overriden? If not I ve miss something in the changelog that change the data file parser. In this case you using the --rebuild option might solves the issue.

mrv84 commented 8 years ago

I just verified and the squidanalyzer.conf it's correct. Already used the -r option, now users and topdenied menu are empty

darold commented 8 years ago

I'm not able to reproduce the issue. In whitch view (year, month, or day views) these menu are empty? Are you using a squidguard log as input?

mrv84 commented 8 years ago

My squidanalyzer.conf

Set the path to the Squid log file LogFile /var/log/squid/access.log /var/log/squidGuard/squid-bl.log

I verified and users, top denied, top domain menu are empty on every views (years, month and day)

tierpod commented 8 years ago

@mrv84 Can you modify squidanalyzer.conf option QuietMode to 0, run

squid-analyzer -r -d

and paste output to pastebin.com?

mrv84 commented 8 years ago

@tierpod

http://pastebin.com/VL777JXx

darold commented 8 years ago

There's nothing wrong in your debug report. Please execute the following commands:

squid-analyzer -d

to proceed last log entries and paste output to pastebin.com together with the output of command:

cat /var/log/squid/access.log /var/log/squidGuard/squid-bl.log

I need to see if the parsing offset for both files have the right values.

Thanks,

mrv84 commented 8 years ago

Hi, sorry for the late answer

########################

[root@localhost ~]# squid-analyzer -d SquidAnalyzer version 6.2 ERROR: you must give a valid path to the Squid log file.

Here the result of /var/log/squid/access.log /var/log/squidGuard/squid-bl.log

http://pastebin.com/qMn1k4nJ

Thanks for your support

mrv84 commented 8 years ago

@darold Any news for me?

darold commented 8 years ago

Please paste the output of:

squid-analyzer -d /var/log/squid/access.log /var/log/squidGuard/squid-bl.log

I was thinking that you had set the logfile in the LogFile configuration directive, but I see that you are using command line arguments.

mrv84 commented 8 years ago

Hi @darold I had set the logfile in the LogFile configuration directive. as indicated on my previous post

mrv84 commented on 23 Sep My squidanalyzer.conf

Set the path to the Squid log file LogFile /var/log/squid/access.log /var/log/squidGuard/squid-bl.log

here the result of : squid-analyzer -d /var/log/squid/access.log /var/log/squidGuard/squid-bl.log

http://pastebin.com/QHwWBN69

darold commented 8 years ago

Ok, I don't see any problem other than what seems a wrong offset in your /var/log/squidGuard/squid-bl.log. To solve that, remove the /var/www/squidanalyzer/SquidGuard.current history file then run squid-analyzer again.

And please download latest release from https://sourceforge.net/projects/squid-report/files/squid-report/6.3/ and install it.

mrv84 commented 8 years ago

@darold followed your instructions Updated to 6.3 Deleted .current file re-executed the squid-analyzer command

squid-analyzer -d /var/log/squid/access.log /var/log/squidGuard/squid-bl.log

Still getting No data on Users and Top Denied Menu. Top Urls and Top Domains were correctly populated

darold commented 8 years ago

I don't understand what's wrong with your installation. If possible can you create an archive with the following content:

tar cjf sareport-mrv84.tar.bz2 /var/www/squidanalyzer/ /var/log/squid/access.log /var/log/squidGuard/squid-bl.log

and send to my privarte email gilles [at] darold [dot] net a link to download the archive.

Regards,