search

msimerson / log-ship-elastic-postfix

Normalize Postfix log messages and save them to Elasticsearch
https://www.npmjs.com/package/log-ship-elastic-postfix
29 stars 12 forks source link

Request to add support for adding rspamd/rmilter filtering #24

Open geosone opened 7 years ago

geosone commented 7 years ago

This log shipper is great. it would solve all our problems if it would also add the spamd/rmilter data

here an example the rspamd and rmilter have a sessiion id and in one oft the loglines thare is the postfix queue ID so the complete rspamd and rmilter sessin can be related to the postfix session. rmilter and rspamd can appear more then one (with different session id's) time like in this example.

`

<22>Mar 9 12:53:44 pro-mailer01-v postfix-In/smtpd[24583]: lost connection after CONNECT from unknown[192.168.202.252] <22>Mar 9 12:53:44 pro-mailer01-v postfix-In/smtpd[24583]: disconnect from unknown[192.168.202.252] <22>Mar 9 12:53:45 pro-mailer01-v postfix-In/smtpd[24587]: connect from unknown[192.168.202.98] <22>Mar 9 12:53:45 pro-mailer01-v rmilter[765]: <8d3d18f36d>; accepted connection from in.mail.sms.at; client: 192.168.202.98:43220 ([192.168.202.98]) <22>Mar 9 12:53:45 pro-mailer01-v postfix-In/smtpd[24587]: 795941FED7: client=unknown[192.168.202.98] <22>Mar 9 12:53:45 pro-mailer01-v rmilter[765]: <8d3d18f36d>; mlfi_data: queue id: <795941FED7> <22>Mar 9 12:53:45 pro-mailer01-v postfix-In/cleanup[26998]: 795941FED7: message-id=<58c14249.3tTSz/XetnnXQjtn%mario.fetka@gmail.com> <22>Mar 9 12:53:45 pro-mailer01-v rmilter[765]: <8d3d18f36d>; mlfi_eom: tempfile=/tmp/msg.XXaeOaSj, size=587 <22>Mar 9 12:53:45 pro-mailer01-v rmilter[765]: <8d3d18f36d>; spamdscan: start scanning message on 192.168.20.99 <30>Mar 9 12:53:45 pro-mailer01-v rspamd[14999]: <297207>; task; accept_socket: accepted connection from 192.168.20.99 port 33191, task ptr: 00007F5B2CB6E010 <30>Mar 9 12:53:45 pro-mailer01-v rspamd[14999]: <297207>; task; rspamd_message_parse: loaded message; id: <58c14249.3tTSz/XetnnXQjtn%mario.fetka@gmail.com>; queue-id: <795941FED7>; size: 587; checksum: <30>Mar 9 12:53:45 pro-mailer01-v rspamd[14999]: <297207>; lua; settings.lua:249: check for settings <22>Mar 9 12:53:45 pro-mailer01-v postfix-In/smtpd[24579]: connect from unknown[unknown] <30>Mar 9 12:53:46 pro-mailer01-v rspamd[14999]: <297207>; task; spf_symbol_callback: skip SPF checks for local networks and authorized users <30>Mar 9 12:53:46 pro-mailer01-v rspamd[14999]: <297207>; task; fuzzy_generate_commands: <58c14249.3tTSz/XetnnXQjtn%mario.fetka@gmail.com>, part is shorter than 100 bytes (71 bytes), skip fuzzy check <30>Mar 9 12:53:46 pro-mailer01-v rspamd[14999]: <297207>; task; dkim_symbol_callback: skip DKIM checks for local networks and authorized users <30>Mar 9 12:53:46 pro-mailer01-v rspamd[14999]: <297207>; lua; once_received.lua:71: Skipping once_received for authenticated user or local network <30>Mar 9 12:53:46 pro-mailer01-v rspamd[14999]: <297207>; lua; dmarc.lua:101: skip DMARC checks for local networks and authorized users <30>Mar 9 12:53:46 pro-mailer01-v rspamd[14999]: <297207>; task; bayes_classify: skip classification as ham class has not enough learns: 0, 200 required <30>Mar 9 12:53:46 pro-mailer01-v rspamd[14999]: <297207>; task; rspamd_task_write_log: id: <58c14249.3tTSz/XetnnXQjtn%mario.fetka@gmail.com>, qid: <795941FED7>, ip: 192.168.202.98, from: , (default: F (no action): [0.90/15.00] [MID_CONTAINS_FROM(1.00){},MIME_GOOD(-0.10){text/plain;},FROM_EQ_ENVFROM(0.00){},FROM_HAS_DN(0.00){},RCPT_COUNT_1(0.00){},RCVD_COUNT_2(0.00){},TO_DN_ALL(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]), len: 587, time: 628.012ms real, 3.037ms virtual, dns req: 0, digest: , rcpts: , mime_rcpt: <30>Mar 9 12:53:46 pro-mailer01-v rspamd[14999]: <297207>; task; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 1 regexps matched, 272 regexps total, 103 regexps cached, 0B bytes scanned using pcre, 754B bytes scanned total <22>Mar 9 12:53:46 pro-mailer01-v rmilter[765]: <8d3d18f36d>; spamdscan: finish scanning message on 192.168.20.99 <22>Mar 9 12:53:46 pro-mailer01-v rmilter[765]: <8d3d18f36d>; spamdscan: scan, time: 0.642, server: 192.168.20.99, metric: default: [0.900 / 15.000], symbols: MIME_GOOD(-0.10)[text/plain], MID_CONTAINS_FROM(1.00)[], TO_DN_ALL(0.00)[], TO_MATCH_ENVRCPT_ALL(0.00)[], RCVD_COUNT_2(0.00)[], FROM_EQ_ENVFROM(0.00)[], FROM_HAS_DN(0.00)[], RCPT_COUNT_1(0.00)[] <22>Mar 9 12:53:46 pro-mailer01-v rmilter[765]: <8d3d18f36d>; clamscan: start scanning message on /var/run/clamav/clamd.ctl <22>Mar 9 12:53:46 pro-mailer01-v rmilter[765]: <8d3d18f36d>; clamscan: finish scanning message on /var/run/clamav/clamd.ctl <22>Mar 9 12:53:46 pro-mailer01-v rmilter[765]: <8d3d18f36d>; clamscan: scan 0.002353, /var/run/clamav/clamd.ctl, /tmp/msg.XXaeOaSj <22>Mar 9 12:53:46 pro-mailer01-v rmilter[765]: <8d3d18f36d>; msg done: queue_id: <795941FED7>; message id: <58c14249.3tTSz/XetnnXQjtn%mario.fetka@gmail.com>; ip: 192.168.202.98; from: ; rcpt: (1 total); user: unauthorized; spam scan: no spam; virus scan: clean; dkim: not signed, ignored <22>Mar 9 12:53:46 pro-mailer01-v postfix-In/qmgr[10245]: 795941FED7: from=, size=731, nrcpt=1 (queue active) <22>Mar 9 12:53:46 pro-mailer01-v postfix-In/smtpd[24587]: disconnect from unknown[192.168.202.98] <22>Mar 9 12:53:46 pro-mailer01-v postfix-In/local[27000]: 795941FED7: to=, relay=local, delay=0.85, delays=0.77/0.05/0/0.02, dsn=2.0.0, status=sent (delivered to command: /usr/local/bin/enqueue-mail -q1 /usr/local/spool/email_in_001 -q2 /usr/local/spool/email_in_002 -q3 /usr/local/spool/email_in_003 -q4 /usr/local/spool/email_in_004 -u 2264103) <22>Mar 9 12:53:46 pro-mailer01-v postfix-In/qmgr[10245]: 795941FED7: removed `
msimerson commented 7 years ago

This module receives maintenance when I need it (ie, not very often). Which means, I won't likely address this until I'm working on this module for another reason. If being very patient doesn't work, your choices are:

  1. submit a PR
  2. post a bounty