search

darold / sendmailanalyzer

Sendmail log Analyzer is a tool to monitor sendmail usage and generate HTML and graph reports. It reports all you ever wanted to know about email trafic on your network. You can also use it in ISP environment with per domain and per mailbox report.
http://sendmailanalyzer.darold.net/
GNU General Public License v3.0
76 stars 13 forks source link

data files missing for Virus and Milter #38

Closed enekux closed 8 years ago

enekux commented 8 years ago

Hi there,

first of all, I am very impressed with your software it looks great!!

I am having problems processing some data.

When in the web frontend I check under "AntiSpam details" and click for example on "Amavis" I see a message "No record for this period".

Then in cli when I check the data files here:

ls -la /usr/local/sendmailanalyzer/data/goban/2016/04/21/ total 29540 drwxr-sr-x 2 root staff 4096 Apr 21 13:50 . drwxr-sr-x 3 root staff 4096 Apr 21 13:50 .. -rw-r--r-- 1 root staff 181406 Apr 21 14:55 dnsbl.dat -rw-r--r-- 1 root staff 2695 Apr 21 14:55 dsn.dat -rw-r--r-- 1 root staff 22957161 Apr 21 14:55 other.dat -rw-r--r-- 1 root staff 1851746 Apr 21 13:50 postgrey.dat -rw-r--r-- 1 root staff 1340382 Apr 21 14:55 recipient.dat -rw-r--r-- 1 root staff 1290418 Apr 21 14:55 rejected.dat -rw-r--r-- 1 root staff 1932148 Apr 21 14:55 senders.dat -rw-r--r-- 1 root staff 343786 Apr 21 14:55 spam.dat -rw-r--r-- 1 root staff 289446 Apr 21 14:55 syserr.dat

I noticed that the next data files are missing:

virus.dat: viruses informations. miltername.dat: message related to a milter, antivir or antispam.

At the conf file I see that for example Amavis I have the next parameter:

grep -B 2 AMAVIS /usr/local/sendmailanalyzer/sendmailanalyzer.conf Syslog name of Amavis. Syslog write it to maillog with the pid as follow: ... amavis[1234] ... This is required to only parse relevant logged lines AMAVIS_NAME amavis|maiad

I wonder if your application is missing the logs since amavis is writting logs with the next format?

/usr/sbin/amavisd-new[3403]:

Any ideas?

Thank you,

enekux commented 8 years ago

Hi again,

I have noticed that I am facing similar problem described in Closed case #26.

My amavis logs are like that:

May 3 16:05:18 mx /usr/sbin/amavisd-new[13711]: (13711-09) Passed INFECTED (Eicar-Test-Signature) {RelayedTaggedInbound,Quarantined}, [2a01:5e0:36:5001::1491:8ce5]:53765 [2a01:5e0:36:5001::1491:8ce5] -> , quarantine: q/virus-qJA8TcS_yq8O, Queue-ID: CF90E558, Message-ID: 20160503141422.682BC6FAE8@emkei.cz, mail_id: qJA8TcS_yq8O, Hits: -1.106, size: 1206, queued_as: E4491A1E, 1056 ms

May 3 16:15:03 mx /usr/sbin/amavisd-new[22538]: (22538-01) Passed CLEAN {RelayedInbound}, -> , Message-ID: 20160503141501.D10A9A88@mx, mail_id: Ropcj22zDmeo, Hits: -0.002, size: 356, queued_as: DA3696CE, 1150 ms

May 3 08:19:25 mx /usr/sbin/amavisd-new[20750]: (20750-01-9) Passed SPAMMY {RelayedTaggedInbound}, [85.206.172.61]:43780 [85.206.172.61] -> , Queue-ID: C0E10CDC5, Message-ID: <20160503141329.25E9506 2762AF83D@mx>, mail_id: B82aODt4cl7s, Hits: 6.951, size: 1904, queued_as: A00B3CD24, 46805 ms

any help please?

darold commented 8 years ago

Hi,

Thanks for the report, you have to change the AMAVIS_NAME configuration directive as follow:

AMAVIS_NAME     \/usr\/sbin\/amavisd-new|amavis|maiad

Commit e43a96c change the default value in configuration file.

enekux commented 8 years ago

Hi thank you for the prompt answer.

I have added the directive and now amavisd-new logs are being parsed but I this has created another issue. When I access to the web portal's main site it shows me this:

Consult global statistics for 'mx.domain.com' Consult global statistics for 'mx'

so I see under the data directory (/usr/local/sendmailanalyzer/data/) a new folder with fqdn...

In the new data folder with fqdn "amavis.dat" data file is shown but others are missing like "dnsbl.dat"

Any ideas?

Thank you,

PS: Please double check the commit because it differs from the line shown here in the forum...

In the commit I see 2 backlashes for escape...

darold commented 8 years ago

The 2 backslashes are normal, when used in a perl string backslashes need to be escaped too.

Your problem comes from the 4th field of the log line, the ident name. Amavis used mx and Postfix use the FQDN. I can add an option to remove the domain part of the ident name but it might be possible to fix that at system side.

enekux commented 8 years ago

hmm...

In my case Amavis uses FQDN and Postfix hostname

May 4 11:17:15 mx.domain /usr/sbin/amavisd-new[18452]: (18452-01) FWD from user@domain -> user@domain,BODY=7BIT 250 2.0.0 from MTA(smtp:[::1]:10025): 250 2.0.0 Ok: queued as 6D49A9DC May 4 11:17:15 mx postfix/lmtp[11420]: 62D9CB6A: to=user@domain, relay=127.0.0.1[127.0.0.1]:10026, delay=1.1, delays=0/0/0/1.1, dsn=2.0.0, status=sent (250 2.0.0 from MTA(smtp:[::1]:10025): 250 2.0.0 Ok: queued as 6D49A9DC)

I have tried to remove the domain part in Amavis configuration but it is not possible.

In Postfix I don't know how to add the domain to the 4th field...

darold commented 8 years ago

Last commit bb2a958 adds the NO_HOST_DOMAIN configuration directive. When activated, remove domain part of the syslog hostname. So in you case after upgrade to latest development code, place the following at bottom of your sendmailanalyzer.conf file:

NO_HOST_DOMAIN        1

Let me know

enekux commented 8 years ago

not working...

darold commented 8 years ago

Can you send me some Kb of your log file to my private email?

enekux commented 8 years ago

Hi, after building from scratch with latest commit it is working OK now.

Thank you very much for your help.

darold commented 8 years ago

OK, good news. Let me know if there's any other issue.