search

NethServer / dev

NethServer issue tracker
https://github.com/NethServer/dev/issues
63 stars 20 forks source link

Upgrade to Rspamd 2.x #5964

Closed stephdl closed 4 years ago

stephdl commented 4 years ago

Go to rspamd 2.x (at the time 2.2)

Proposed solution make rspamd2.x compatible with our configuration

PR: https://github.com/NethServer/nethserver-mail/pull/155

other issue in relation : https://github.com/NethServer/dev/issues/5940

See also

https://community.nethserver.org/t/rspamd-update-request/13937

thank xalex77

DavidePrincipi commented 4 years ago

Added rspamd-2.2-1.x86_64.rpm to testing repository.

nethbot commented 4 years ago

in 7.7.1908/testing:

stephdl commented 4 years ago

QA

  1. antivirus:

    • check virus is rejected
    • check if clamav is not reachable, then a softreject is done (try again later) : this has not be modified by our PR but it is consistent with antivirus
    • for authenticated users (from webmail) the softreject must not occur) : this has not be modified by our PR but it is consistent with antivirus

  2. Reputation is a new module you will see some new symbol in log

R_DKIM_ALLOW
R_DKIM_REJECT
R_SPF_ALLOW
R_SPF_REJECT
IP_REPUTATION_HAM
IP_REPUTATION_SPAM
GENERIC_REPUTATION

this will examine now the reputation and add or remove score to the email, it is something new and quite accurate, only you have to check them

  1. bayes filter bayes was not really workable with 1.9.x the stats number was not the same following you learn by 
rspamc # dovecot : move spam to junk
web ui 
autolearn

now when you check the log you must find this, whatever the service rspamd restart or not

95030:Dec 11 09:40:04 prometheus rspamd[10578]: <7dc781>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 140; 200 required
95031:Dec 11 09:40:04 prometheus rspamd[10578]: <7dc781>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 112; 200 required
95065:Dec 11 09:40:58 prometheus rspamd[10578]: <8ebbc7>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 112; 200 required
95066:Dec 11 09:40:58 prometheus rspamd[10578]: <8ebbc7>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 140; 200 required
95090:Dec 11 09:42:05 prometheus rspamd[10578]: <8b09d9>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 140; 200 required
95091:Dec 11 09:42:05 prometheus rspamd[10578]: <8b09d9>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 112; 200 required
95093:Dec 11 09:42:05 prometheus rspamd[10578]: <8b09d9>; proxy; rspamd_stat_check_autolearn: <undef>: autolearn spam for classifier 'bayes' as message's action is reject, score: 24.69
95110:Dec 11 09:45:24 prometheus rspamd[10578]: <781178>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 141; 200 required
95111:Dec 11 09:45:24 prometheus rspamd[10578]: <781178>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 112; 200 required
95135:Dec 11 09:45:48 prometheus rspamd[10578]: <0fd7a3>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 141; 200 required
95136:Dec 11 09:45:48 prometheus rspamd[10578]: <0fd7a3>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 112; 200 required

as you can see the autolearn has increased the counter

autolearn work as is , score <-1 it is a ham, rejected because the score is > SpamKillLevel (default 19.9) it is spam

you must check that autolearn works automatically, with the move to junk folder and with the rspamd UI

  1. whitelist/blacklist has been modified recently by another PR to rsapmd 1.9.4, you should check that the QA is still good
stephdl commented 4 years ago

QA

4 (WL/BL)

workable

nethbot commented 4 years ago

in 7.7.1908/testing:

stephdl commented 4 years ago

whats up @davidep on your email server upgraded to rspamd 2

DavidePrincipi commented 4 years ago

Test case 1 - FAILED (?)

check if clamav is not reachable, then a softreject is done (try again later) : this has not be modified by our PR but it is consistent with antivirus

If I systemctl stop clamd@rspamd a message with Eicar attached is accepted. Mail is sent from SMTP authenticated first.user to second.user on the same system.

In /var/log/messages:

Jan 14 18:32:27 vm5 postfix/smtpd[20163]: 4915948739B: client=gateway[192.168.122.1], sasl_method=PLAIN, sasl_username=first.user@dpnet.nethesis.it
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; milter; rspamd_milter_process_command: got connection from 192.168.122.1:59484
Jan 14 18:32:27 vm5 postfix/cleanup[20274]: 4915948739B: message-id=<6b383e1d-f964-5770-2821-98b1f9780d4a@dpnet.nethesis.it>
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_message_parse: loaded message; id: <6b383e1d-f964-5770-2821-98b1f9780d4a@dpnet.nethesis.it>; queue-id: <4915948739B>; size: 1227; checksum: <56dcb06ecdafd99848c9a8f2ae43bb6f>
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; settings.lua:324: <6b383e1d-f964-5770-2821-98b1f9780d4a@dpnet.nethesis.it> apply static settings authenticated (id = 1937017268); authenticated matched
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; asn.lua:76: cannot query ip 1.122.168.192.asn.rspamd.com on 127.0.0.1: no results
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_mime_part_detect_language: detected part language: en
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; spf_symbol_callback: skip SPF checks for local networks and authorized users
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; dkim_symbol_callback: skip DKIM checks for local networks and authorized users
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; dmarc.lua:572: skip DMARC checks for local networks and authorized users
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; mime_types.lua:293: using special tables from user settings
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; once_received.lua:98: Skipping once_received for authenticated user or local network
Jan 14 18:32:27 vm5 rspamd[6795]: <>; ; rspamd_inet_address_connect: connect unix:/var/run/clamd@rspamd/clamav failed: 2, 'No such file or directory'
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; lua_tcp_make_connection: cannot connect to /var/run/clamd@rspamd/clamav (/var/run/clamd@rspamd/clamav): No such file or directory
Jan 14 18:32:27 vm5 rspamd[6795]: <>; ; rspamd_inet_address_connect: connect unix:/var/run/clamd@rspamd/clamav failed: 2, 'No such file or directory'
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; lua_tcp_make_connection: cannot connect to /var/run/clamd@rspamd/clamav (/var/run/clamd@rspamd/clamav): No such file or directory
Jan 14 18:32:27 vm5 rspamd[6795]: <>; ; rspamd_inet_address_connect: connect unix:/var/run/clamd@rspamd/clamav failed: 2, 'No such file or directory'
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; lua_tcp_make_connection: cannot connect to /var/run/clamd@rspamd/clamav (/var/run/clamd@rspamd/clamav): No such file or directory
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; clamav.lua:119: clamav: failed to scan, maximum retransmits exceed
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; lua; common.lua:107: clamav: result - FAILED with error: "failed to scan and retransmits exceed - score: 0"
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_SPAM of classifier bayes: not enough learns 0; 200 required
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_redis_connected: skip obtaining bayes tokens for BAYES_HAM of classifier bayes: not enough learns 0; 200 required
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_stat_classifiers_process: skip statistics as SPAM class is missing
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_task_write_log: id: <6b383e1d-f964-5770-2821-98b1f9780d4a@dpnet.nethesis.it>, qid: <4915948739B>, ip: 192.168.122.1, user: first.user@dpnet.nethesis.it, from: <first.user@dpnet.nethesis.it>, (default: F (no action): [2.00/20.00] [CTYPE_MIXED_BOGUS(1.00){},MIME_BASE64_TEXT_BOGUS(1.00){},MIME_BASE64_TEXT(0.10){},MIME_GOOD(-0.10){multipart/mixed;text/plain;},CLAM_VIRUS_FAIL(0.00){failed to scan and retransmits exceed;},FROM_EQ_ENVFROM(0.00){},FROM_HAS_DN(0.00){},HAS_ATTACHMENT(0.00){},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;1:+;2:+;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_ZERO(0.00){0;},TO_DN_NONE(0.00){},TO_DOM_EQ_FROM_DOM(0.00){},TO_MATCH_ENVRCPT_ALL(0.00){}]), len: 1227, time: 47.983ms, dns req: 1, digest: <56dcb06ecdafd99848c9a8f2ae43bb6f>, rcpts: <second.user@dpnet.nethesis.it>, mime_rcpts: <second.user@dpnet.nethesis.it>, settings_id: authenticated
Jan 14 18:32:27 vm5 rspamd[6795]: <802641>; proxy; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 6 regexps matched, 184 regexps total, 99 regexps cached, 0B scanned using pcre, 1.28KiB scanned total
Jan 14 18:32:27 vm5 opendkim[6943]: 4915948739B: no signing table match for 'first.user@dpnet.nethesis.it'
Jan 14 18:32:27 vm5 postfix/qmgr[7063]: 4915948739B: from=<first.user@dpnet.nethesis.it>, size=1518, nrcpt=1 (queue active)
Jan 14 18:32:27 vm5 postfix/smtpd[20163]: disconnect from gateway[192.168.122.1]
Jan 14 18:32:27 vm5 rspamd[6795]: <81aa1a>; proxy; proxy_milter_finish_handler: finished milter connection
Jan 14 18:32:27 vm5 dovecot: lmtp(20507): Connect from local
Jan 14 18:32:27 vm5 dovecot: lmtp(second.user@dpnet.nethesis.it): WIzvFyv7HV4bUAAAXdYZ9Q: sieve: msgid=<6b383e1d-f964-5770-2821-98b1f9780d4a@dpnet.nethesis.it>: stored mail into mailbox 'INBOX'
Jan 14 18:32:27 vm5 dovecot: lmtp(20507): Disconnect from local: Successful quit
Jan 14 18:32:27 vm5 postfix/lmtp[20506]: 4915948739B: to=<second.user@dpnet.nethesis.it>, relay=vm5.dpnet.nethesis.it[/var/run/dovecot/lmtp], delay=0.12, delays=0.06/0.03/0.02/0.02, dsn=2.0.0, status=sent (250 2.0.0 <second.user@dpnet.nethesis.it> WIzvFyv7HV4bUAAAXdYZ9Q Saved)
Jan 14 18:32:27 vm5 postfix/qmgr[7063]: 4915948739B: removed

@stephdl, can you reproduce it?

stephdl commented 4 years ago

it is normal, we decided to not reject for authenticated user

/etc/rspamd/local.d/settings.conf
#Do not soft reject if clamav is not reachable

authenticated {
        priority = high;
        authenticated = yes;
        apply {
                symbols_disabled = ["FORCE_ACTION_CLAM_VIRUS_FAIL"];
        }
}
stephdl commented 4 years ago

it works on my system 


[root@prometheus ~]# grep -srni 'FORCE_ACTION_CLAM_VIRUS_FAIL' /var/log/maillog
7616:Jan 12 23:27:10 prometheus rspamd[3818]: <b7fde1>; proxy; rspamd_task_write_log: id: <20200112222639.541D830319DBF@NS7.stephdl.dynu.net>, qid: <14C4C18CE1DA4>, ip: 217.72.192.75, from: <no-reply@NS7.stephdl.dynu.net>, (default: F (soft reject): [0.40/19.90] [MX_INVALID(0.50){cached;},MIME_GOOD(-0.10){text/plain;},ASN(0.00){asn:8560, ipnet:217.72.192.0/20, country:DE;},CLAM_VIRUS_FAIL(0.00){failed to scan and retransmits exceed;},DMARC_NA(0.00){stephdl.dynu.net;},FORCE_ACTION_CLAM_VIRUS_FAIL(0.00){soft reject;},FROM_EQ_ENVFROM(0.00){},FROM_HAS_DN(0.00){},GENERIC_REPUTATION(0.00){-0.43200329875004;},IP_REPUTATION_HAM(0.00){asn: 8560(-0.12), country: DE(-0.00), ip: 217.72.192.75(-0.43);},MID_RHS_MATCH_FROM(0.00){},MIME_TRACE(0.00){0:+;},RCPT_COUNT_THREE(0.00){3;},RCVD_COUNT_TWO(0.00){2;},RCVD_IN_DNSWL_NONE(0.00){217.72.192.75:from;},RCVD_TLS_ALL(0.00){},RCVD_VIA_SMTP_AUTH(0.00){},RECEIVED_SPAMHAUS_PBL(0.00){86.195.248.166:received;},R_DKIM_NA(0.00){},R_SPF_NA(0.00){},TO_DN_NONE(0.00){},TO_MATCH_ENVRCPT_SOME(0.00){}]), len: 4735, time: 15580.754ms, dns req: 33, digest: <61b4552f791e10fb453be73879b0935f>, rcpts: <admin@de-labrusse.fr>, mime_rcpts: <admin@de-labrusse.fr,plop@plop.com,root@NS7.stephdl.dynu.net>, forced: soft reject "Cannot validate the message now. Try again later"; score=nan (set by force_actions)
DavidePrincipi commented 4 years ago

TEST case 1 - VERIFIED

Now authenticated users can skip the AV check if clamd is not responsive. MTAs get back a soft reject instead.

QA tricks:

Packager note

Release also olefy-0.55

nethbot commented 4 years ago

in 7.7.1908/updates:

nethbot commented 4 years ago

in 7.7.1908/updates:

DavidePrincipi commented 4 years ago

in 7.7.1908/updates: