Closed stephdl closed 4 years ago
Added rspamd-2.2-1.x86_64.rpm
to testing repository.
in 7.7.1908/testing
:
QA
antivirus:
Reputation is a new module you will see some new symbol in log
R_DKIM_ALLOW
R_DKIM_REJECT
R_SPF_ALLOW
R_SPF_REJECT
IP_REPUTATION_HAM
IP_REPUTATION_SPAM
GENERIC_REPUTATION
this will examine now the reputation and add or remove score to the email, it is something new and quite accurate, only you have to check them
rspamc # dovecot : move spam to junk
web ui
autolearn
now when you check the log you must find this, whatever the service rspamd restart or not
95030:Dec 11 09:40:04 prometheus rspamd[10578]: <7dc781>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 140; 200 required
95031:Dec 11 09:40:04 prometheus rspamd[10578]: <7dc781>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 112; 200 required
95065:Dec 11 09:40:58 prometheus rspamd[10578]: <8ebbc7>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 112; 200 required
95066:Dec 11 09:40:58 prometheus rspamd[10578]: <8ebbc7>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 140; 200 required
95090:Dec 11 09:42:05 prometheus rspamd[10578]: <8b09d9>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 140; 200 required
95091:Dec 11 09:42:05 prometheus rspamd[10578]: <8b09d9>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 112; 200 required
95093:Dec 11 09:42:05 prometheus rspamd[10578]: <8b09d9>; proxy; rspamd_stat_check_autolearn: <undef>: autolearn spam for classifier 'bayes' as message's action is reject, score: 24.69
95110:Dec 11 09:45:24 prometheus rspamd[10578]: <781178>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 141; 200 required
95111:Dec 11 09:45:24 prometheus rspamd[10578]: <781178>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 112; 200 required
95135:Dec 11 09:45:48 prometheus rspamd[10578]: <0fd7a3>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 141; 200 required
95136:Dec 11 09:45:48 prometheus rspamd[10578]: <0fd7a3>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 112; 200 required
as you can see the autolearn has increased the counter
autolearn work as is , score <-1
it is a ham, rejected because the score is > SpamKillLevel
(default 19.9) it is spam
you must check that autolearn works automatically, with the move to junk folder and with the rspamd UI
QA
4 (WL/BL)
workable
in 7.7.1908/testing
:
whats up @davidep on your email server upgraded to rspamd 2
Test case 1 - FAILED (?)
check if clamav is not reachable, then a softreject is done (try again later) : this has not be modified by our PR but it is consistent with antivirus
If I systemctl stop clamd@rspamd
a message with Eicar attached is accepted. Mail is sent from SMTP authenticated first.user to second.user on the same system.
In /var/log/messages
:
Jan 14 18:32:27 vm5 postfix/smtpd[20163]: 4915948739B: client=gateway[192.168.122.1], sasl_method=PLAIN, sasl_username=first.user@dpnet.nethesis.it
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; milter; rspamd_milter_process_command: got connection from 192.168.122.1:59484
Jan 14 18:32:27 vm5 postfix/cleanup[20274]: 4915948739B: message-id=<6b383e1d-f964-5770-2821-98b1f9780d4a@dpnet.nethesis.it>
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_message_parse: loaded message; id: <6b383e1d-f964-5770-2821-98b1f9780d4a@dpnet.nethesis.it>; queue-id: <4915948739B>; size: 1227; checksum: <56dcb06ecdafd99848c9a8f2ae43bb6f>
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; settings.lua:324: <6b383e1d-f964-5770-2821-98b1f9780d4a@dpnet.nethesis.it> apply static settings authenticated (id = 1937017268); authenticated matched
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; asn.lua:76: cannot query ip 1.122.168.192.asn.rspamd.com on 127.0.0.1: no results
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_mime_part_detect_language: detected part language: en
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; spf_symbol_callback: skip SPF checks for local networks and authorized users
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; dkim_symbol_callback: skip DKIM checks for local networks and authorized users
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; dmarc.lua:572: skip DMARC checks for local networks and authorized users
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; mime_types.lua:293: using special tables from user settings
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; once_received.lua:98: Skipping once_received for authenticated user or local network
Jan 14 18:32:27 vm5 rspamd[6795]: <>; ; rspamd_inet_address_connect: connect unix:/var/run/clamd@rspamd/clamav failed: 2, 'No such file or directory'
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; lua_tcp_make_connection: cannot connect to /var/run/clamd@rspamd/clamav (/var/run/clamd@rspamd/clamav): No such file or directory
Jan 14 18:32:27 vm5 rspamd[6795]: <>; ; rspamd_inet_address_connect: connect unix:/var/run/clamd@rspamd/clamav failed: 2, 'No such file or directory'
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; lua_tcp_make_connection: cannot connect to /var/run/clamd@rspamd/clamav (/var/run/clamd@rspamd/clamav): No such file or directory
Jan 14 18:32:27 vm5 rspamd[6795]: <>; ; rspamd_inet_address_connect: connect unix:/var/run/clamd@rspamd/clamav failed: 2, 'No such file or directory'
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; lua_tcp_make_connection: cannot connect to /var/run/clamd@rspamd/clamav (/var/run/clamd@rspamd/clamav): No such file or directory
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; clamav.lua:119: clamav: failed to scan, maximum retransmits exceed
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; common.lua:107: clamav: result - FAILED with error: "failed to scan and retransmits exceed - score: 0"
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 0; 200 required
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 0; 200 required
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_task_write_log: id: <6b383e1d-f964-5770-2821-98b1f9780d4a@dpnet.nethesis.it>, qid: <4915948739B>, ip: 192.168.122.1, user: first.user@dpnet.nethesis.it, from: <first.user@dpnet.nethesis.it>, (default: F (no action): [2.00/20.00] [CTYPE_MIXED_BOGUS(1.00){},MIME_BASE64_TEXT_BOGUS(1.00){},MIME_BASE64_TEXT(0.10){},MIME_GOOD(-0.10){multipart/mixed;text/plain;},CLAM_VIRUS_FAIL(0.00){failed to scan and retransmits exceed;},FROM_EQ_ENVFROM(0.00){},FROM_HAS_DN(0.00){},HAS_ATTACHMENT(0.00){},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;1:+;2:+;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_ZERO(0.00){0;},TO_DN_NONE(0.00){},TO_DOM_EQ_FROM_DOM(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]), len: 1227, time: 47.983ms, dns req: 1, digest: <56dcb06ecdafd99848c9a8f2ae43bb6f>, rcpts: <second.user@dpnet.nethesis.it>, mime_rcpts: <second.user@dpnet.nethesis.it>, settings_id: authenticated
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 6 regexps matched, 184 regexps total, 99 regexps cached, 0B scanned using pcre, 1.28KiB scanned total
Jan 14 18:32:27 vm5 opendkim[6943]: 4915948739B: no signing table match for 'first.user@dpnet.nethesis.it'
Jan 14 18:32:27 vm5 postfix/qmgr[7063]: 4915948739B: from=<first.user@dpnet.nethesis.it>, size=1518, nrcpt=1 (queue active)
Jan 14 18:32:27 vm5 postfix/smtpd[20163]: disconnect from gateway[192.168.122.1]
Jan 14 18:32:27 vm5 rspamd[6795]: <81aa1a>; proxy; proxy_milter_finish_handler: finished milter connection
Jan 14 18:32:27 vm5 dovecot: lmtp(20507): Connect from local
Jan 14 18:32:27 vm5 dovecot: lmtp(second.user@dpnet.nethesis.it): WIzvFyv7HV4bUAAAXdYZ9Q: sieve: msgid=<6b383e1d-f964-5770-2821-98b1f9780d4a@dpnet.nethesis.it>: stored mail into mailbox 'INBOX'
Jan 14 18:32:27 vm5 dovecot: lmtp(20507): Disconnect from local: Successful quit
Jan 14 18:32:27 vm5 postfix/lmtp[20506]: 4915948739B: to=<second.user@dpnet.nethesis.it>, relay=vm5.dpnet.nethesis.it[/var/run/dovecot/lmtp], delay=0.12, delays=0.06/0.03/0.02/0.02, dsn=2.0.0, status=sent (250 2.0.0 <second.user@dpnet.nethesis.it> WIzvFyv7HV4bUAAAXdYZ9Q Saved)
Jan 14 18:32:27 vm5 postfix/qmgr[7063]: 4915948739B: removed
@stephdl, can you reproduce it?
it is normal, we decided to not reject for authenticated user
/etc/rspamd/local.d/settings.conf
#Do not soft reject if clamav is not reachable
authenticated {
priority = high;
authenticated = yes;
apply {
symbols_disabled = ["FORCE_ACTION_CLAM_VIRUS_FAIL"];
}
}
it works on my system
[root@prometheus ~]# grep -srni 'FORCE_ACTION_CLAM_VIRUS_FAIL' /var/log/maillog
7616:Jan 12 23:27:10 prometheus rspamd[3818]: <b7fde1>; proxy; rspamd_task_write_log: id: <20200112222639.541D830319DBF@NS7.stephdl.dynu.net>, qid: <14C4C18CE1DA4>, ip: 217.72.192.75, from: <no-reply@NS7.stephdl.dynu.net>, (default: F (soft reject): [0.40/19.90] [MX_INVALID(0.50){cached;},MIME_GOOD(-0.10){text/plain;},ASN(0.00){asn:8560, ipnet:217.72.192.0/20, country:DE;},CLAM_VIRUS_FAIL(0.00){failed to scan and retransmits exceed;},DMARC_NA(0.00){stephdl.dynu.net;},FORCE_ACTION_CLAM_VIRUS_FAIL(0.00){soft reject;},FROM_EQ_ENVFROM(0.00){},FROM_HAS_DN(0.00){},GENERIC_REPUTATION(0.00){-0.43200329875004;},IP_REPUTATION_HAM(0.00){asn: 8560(-0.12), country: DE(-0.00), ip: 217.72.192.75(-0.43);},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;},RCPT_COUNT_THREE(0.00){3;},RCVD_COUNT_TWO(0.00){2;},RCVD_IN_DNSWL_NONE(0.00){217.72.192.75:from;},RCVD_TLS_ALL(0.00){},RCVD_VIA_SMTP_AUTH(0.00){},RECEIVED_SPAMHAUS_PBL(0.00){86.195.248.166:received;},R_DKIM_NA(0.00){},R_SPF_NA(0.00){},TO_DN_NONE(0.00){},TO_MATCH_ENVRCPT_SOME(0.00){}]), len: 4735, time: 15580.754ms, dns req: 33, digest: <61b4552f791e10fb453be73879b0935f>, rcpts: <admin@de-labrusse.fr>, mime_rcpts: <admin@de-labrusse.fr,plop@plop.com,root@NS7.stephdl.dynu.net>, forced: soft reject "Cannot validate the message now. Try again later"; score=nan (set by force_actions)
TEST case 1 - VERIFIED
Now authenticated users can skip the AV check if clamd is not responsive. MTAs get back a soft reject instead.
QA tricks:
rs_clamav_*
keys in redis DBPackager note
Release also olefy-0.55
in 7.7.1908/updates
:
in 7.7.1908/updates:
Go to rspamd 2.x (at the time 2.2)
Proposed solution make rspamd2.x compatible with our configuration
PR: https://github.com/NethServer/nethserver-mail/pull/155
other issue in relation : https://github.com/NethServer/dev/issues/5940
See also
https://community.nethserver.org/t/rspamd-update-request/13937
thank xalex77