Upgrade rspamd to 1.7.3

stephdl commented 6 years ago

A new rpm is waiting our testing for rspamd -> 1.7.0

see the changelog https://github.com/vstakhov/rspamd/blob/master/ChangeLog

immediate change

the SYMBOL now goes to /etc/rspamd/score.d
the bayes uses a new format, the training of ham/spam must be done again (a minimu of 200)

feeling

The history tab looks really faster, the history bug probably is fixed

check my PR https://github.com/NethServer/nethserver-mail/pull/27

stephdl commented 6 years ago

the way for now is just to put in a disabled and read-only state. Curl action for saveactions and actions are experimental

nethbot commented 6 years ago

in 7.4.1708/testing:

nethbot commented 6 years ago

in 7.4.1708/testing:

nethbot commented 6 years ago

in 7.4.1708/testing:

stephdl commented 6 years ago

test case @gsanchietti

upgrade to rspamd_1.7.2
score are migrated from override.d/metrics.conf to local.d/actions.conf (template)
threshold are in a disabled state
message to announce that score must be modified in rspamd UI
symbols are now in /etc/rspamd/scores.d/nethserver_group.conf

you can check if the rspamd settings are good before and after the upgrade by rspamadm configdump > file and compare

gsanchietti commented 6 years ago

I've found a couple of issues on testing phase:

metrics.conf isn't deleted after the upgrade (see PR NethServer/nethserver-mail#36)
Some messages from cron, generated from a local server, are now mail marked as SPAM and also have no spam tag added to the header. It seems the mail is marked as SPAM even if total score is 5.90 and threshold is set to 6.

Rspam config:

config show rspamd

rspamd=service
    BlockAttachmentClassList=Exec
    BlockAttachmentCustomList=doc,odt
    BlockAttachmentCustomStatus=disabled
    BlockAttachmentStatus=enabled
    Password=g20ZWQgSKXsOdX_O
    RecipientWhiteList=
    SenderBlackList=root@ns2.itdsolutions.it
    SenderWhiteList=
    SpamCheckStatus=enabled
    SpamGreyLevel=4
    SpamKillLevel=20
    SpamSubjectPrefixStatus=disabled
    SpamSubjectPrefixString=***SPAM***
    SpamTag2Level=6
    VirusAction=reject
    VirusCheckStatus=enabled
    VirusScanOnlyAttachment=false
    VirusScanSize=2000000

cat /etc/rspamd/local.d/actions.conf
...
reject = 20;
add_header = 6;
greylist = 4;
rewrite_subject = null;
subject = "***SPAM*** %s";
...

Header:

Return-Path: <root@birro.nethesis.it>
Received: from nethservice.nethesis.it
    by nethservice.nethesis.it (Dovecot) with LMTP id q2OmFSXLylojXwAAJc5BcA
    ; Mon, 09 Apr 2018 04:08:37 +0200
Received: from birro.nethesis.it (42.nethesis.it [192.168.5.3])
    by nethservice.nethesis.it (Postfix) with ESMTP id 5CCE43088E558;
    Mon,  9 Apr 2018 04:08:36 +0200 (CEST)
Received: by birro.nethesis.it (Postfix)
    id 2F13B1E00AC; Mon,  9 Apr 2018 04:08:36 +0200 (CEST)
Delivered-To: root@birro.nethesis.it
Received: by birro.nethesis.it (Postfix, from userid 0)
    id E700E1E00E0; Mon,  9 Apr 2018 04:08:35 +0200 (CEST)
From: Anacron <root@birro.nethesis.it>
To: root@birro.nethesis.it
Content-Type: text/plain; charset="ANSI_X3.4-1968"
Subject: ***SPAM*** Anacron job 'cron.daily' on birro.nethesis.it
Message-Id: <20180409020835.E700E1E00E0@birro.nethesis.it>
Date: Mon,  9 Apr 2018 04:08:35 +0200 (CEST)

Relevant log part:

Apr  9 04:08:36 nethservice postfix/smtpd[24254]: connect from 42.nethesis.it[192.168.5.3]
Apr  9 04:08:36 nethservice rspamd[19528]: <224741>; proxy; proxy_accept_socket: accepted milter connection from /var/run/rspamd/worker-proxy port 0
Apr  9 04:08:36 nethservice postfix/smtpd[24254]: 5CCE43088E558: client=42.nethesis.it[192.168.5.3]
Apr  9 04:08:36 nethservice rspamd[19528]: <224741>; milter; rspamd_milter_process_command: got connection from 192.168.5.3:47098
Apr  9 04:08:36 nethservice postfix/cleanup[23949]: 5CCE43088E558: message-id=<20180409020835.E700E1E00E0@birro.nethesis.it>
Apr  9 04:08:36 nethservice rspamd[19528]: <224741>; proxy; rspamd_mime_part_get_cte: detected missing CTE for part as: 7bit
Apr  9 04:08:36 nethservice rspamd[19528]: <224741>; proxy; rspamd_extract_words: detected part language: en
Apr  9 04:08:36 nethservice rspamd[19528]: <224741>; proxy; rspamd_message_parse: loaded message; id: <20180409020835.E700E1E00E0@birro.nethesis.it>; queue-id: <5CCE43088E558>; size: 273926; checksum: <f15d1774e
03a3cef985bcdc51d299070>
Apr  9 04:08:36 nethservice rspamd[19528]: <224741>; proxy; dkim_symbol_callback: skip DKIM checks for local networks and authorized users
Apr  9 04:08:36 nethservice rspamd[19528]: <224741>; proxy; spf_symbol_callback: skip SPF checks for local networks and authorized users
Apr  9 04:08:36 nethservice rspamd[19528]: <224741>; lua; once_received.lua:82: Skipping once_received for authenticated user or local network
Apr  9 04:08:36 nethservice rspamd[19528]: <224741>; lua; dmarc.lua:218: skip DMARC checks for local networks and authorized users
Apr  9 04:08:36 nethservice rspamd[19528]: <224741>; lua; ip_score.lua:312: skip IP Score for local networks and authorized users
Apr  9 04:08:37 nethservice rspamd[19528]: <224741>; lua; greylist.lua:268: Downgrading metric action from "greylist" to "no action"
Apr  9 04:08:37 nethservice rspamd[19528]: <224741>; lua; neural.lua:297: ann score: 0.991
Apr  9 04:08:37 nethservice rspamd[19528]: <224741>; lua; replies.lua:113: storing message-id for replies check
Apr  9 04:08:37 nethservice rspamd[19528]: <224741>; proxy; rspamd_task_write_log: id: <20180409020835.E700E1E00E0@birro.nethesis.it>, qid: <5CCE43088E558>, ip: 192.168.5.3, from: <root@birro.nethesis.it>, (defa
ult: F (rewrite subject): [5.90/20.00] [URL_IN_SUBJECT(4.00){0.25;birro.nethesis.it;},FORGED_RECIPIENTS(2.00){},MIME_GOOD(-0.10){text/plain;},FROM_EQ_ENVFROM(0.00){},FROM_HAS_DN(0.00){},MID_RHS_MATCH_FROM(0.00){
},NEURAL_SPAM(0.00){0.991;0;},RCPT_COUNT_ONE(0.00){1;},RCVD_COUNT_THREE(0.00){3;},BAYES_HAM(-0.00){27.96%;},RCVD_NO_TLS_LAST(0.00){},TO_DN_NONE(0.00){},TO_EQ_FROM(0.00){}]), len: 273926, time: 947.892ms real, 11
.328ms virtual, dns req: 9, digest: <f15d1774e03a3cef985bcdc51d299070>, rcpts: <davidep@nethesis.it,filippo@nethesis.it,giacomo@nethesis.it>, mime_rcpt: <root@birro.nethesis.it>
Apr  9 04:08:37 nethservice rspamd[19528]: <224741>; proxy; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 1 regexps matched, 172 regexps total, 89 regexps cached, 0B bytes scanned using 
pcre, 798k bytes scanned total
Apr  9 04:08:37 nethservice postfix/qmgr[19800]: 5CCE43088E558: from=<root@birro.nethesis.it>, size=274192, nrcpt=3 (queue active)
Apr  9 04:08:37 nethservice postfix/smtpd[24254]: disconnect from 42.nethesis.it[192.168.5.3]
Apr  9 04:08:37 nethservice rspamd[19528]: <b83fcc>; proxy; proxy_milter_finish_handler: finished milter connection
Apr  9 04:08:37 nethservice dovecot: lmtp(24355): Connect from local
Apr  9 04:08:37 nethservice dovecot: lmtp(24355, davidep@nethesis.it): copy from <lmtp DATA>: box=junkmail, uid=20197, msgid=<20180409020835.E700E1E00E0@birro.nethesis.it>, from=Anacron <root@birro.nethesis.it>, subject=***SPAM*** Anacron job 'cron.daily' on birro.nethesis.it, flags=()
Apr  9 04:08:37 nethservice dovecot: lmtp(24355, davidep@nethesis.it): q2OmFSXLylojXwAAJc5BcA: sieve: msgid=<20180409020835.E700E1E00E0@birro.nethesis.it>: stored mail into mailbox 'junkmail'
Apr  9 04:08:37 nethservice postfix/lmtp[24354]: 5CCE43088E558: to=<davidep@nethesis.it>, relay=nethservice.nethesis.it[/var/run/dovecot/lmtp], delay=1, delays=0.98/0.01/0/0.03, dsn=2.0.0, status=sent (250 2.0.0 <davidep@nethesis.it> q2OmFSXLylojXwAAJc5BcA Saved)
Apr  9 04:08:37 nethservice dovecot: lmtp(24355, filippo@nethesis.it): copy from <lmtp DATA>: box=junkmail, uid=30623, msgid=<20180409020835.E700E1E00E0@birro.nethesis.it>, from=Anacron <root@birro.nethesis.it>, subject=***SPAM*** Anacron job 'cron.daily' on birro.nethesis.it, flags=()
Apr  9 04:08:37 nethservice dovecot: lmtp(24355, filippo@nethesis.it): q2OmFSXLylojXwAAJc5BcA: sieve: msgid=<20180409020835.E700E1E00E0@birro.nethesis.it>: stored mail into mailbox 'junkmail'
Apr  9 04:08:37 nethservice postfix/lmtp[24354]: 5CCE43088E558: to=<filippo@nethesis.it>, relay=nethservice.nethesis.it[/var/run/dovecot/lmtp], delay=1, delays=0.98/0.01/0/0.04, dsn=2.0.0, status=sent (250 2.0.0 <filippo@nethesis.it> q2OmFSXLylojXwAAJc5BcA Saved)
Apr  9 04:08:37 nethservice dovecot: lmtp(24355, giacomo@nethesis.it): copy from <lmtp DATA>: box=junkmail, uid=1986, msgid=<20180409020835.E700E1E00E0@birro.nethesis.it>, from=Anacron <root@birro.nethesis.it>, subject=***SPAM*** Anacron job 'cron.daily' on birro.nethesis.it, flags=()
Apr  9 04:08:37 nethservice dovecot: lmtp(24355, giacomo@nethesis.it): q2OmFSXLylojXwAAJc5BcA: sieve: msgid=<20180409020835.E700E1E00E0@birro.nethesis.it>: stored mail into mailbox 'junkmail'
Apr  9 04:08:37 nethservice postfix/lmtp[24354]: 5CCE43088E558: to=<giacomo@nethesis.it>, relay=nethservice.nethesis.it[/var/run/dovecot/lmtp], delay=1, delays=0.98/0.01/0/0.05, dsn=2.0.0, status=sent (250 2.0.0 <giacomo@nethesis.it> q2OmFSXLylojXwAAJc5BcA Saved)
Apr  9 04:08:37 nethservice dovecot: lmtp(24355): Disconnect from local: Successful quit
Apr  9 04:08:37 nethservice postfix/qmgr[19800]: 5CCE43088E558: removed

Another mail review wrongly marked as spam:

X-Spam-Flag: Yes
X-Rspamd-Queue-Id: 996AA30858AEC
X-Spamd-Result: default: False [4.48 / 20.00]
     FROM_NEQ_ENVFROM(0.00)[info@dariovignali.net,bounce-150456-1938-157030-alessio.fattorini=nethesis.it@s6.acemsrve.com]
     DMARC_NA(0.00)[dariovignali.net]
     HAS_LIST_UNSUB(-0.01)[]
     R_BAD_CTE_7BIT(3.50)[0.3]
     BAYES_HAM(-0.00)[24.04%]
     FORGED_SENDER_VERP_SRS(0.00)[]
     R_SPF_ALLOW(-0.20)[+ip4:192.92.97.0/24]
     RCVD_COUNT_TWO(0.00)[2]
     DKIM_TRACE(0.00)[dariovignali.net:+]
     PREVIOUSLY_DELIVERED(0.00)[alessio.fattorini@nethesis.it]
     RCVD_NO_TLS_LAST(0.00)[]
     R_DKIM_ALLOW(-0.20)[dariovignali.net]
     ASN(0.00)[asn:32475, ipnet:192.92.97.0/24, country:US]
     FROM_HAS_DN(0.00)[]
     RCPT_COUNT_ONE(0.00)[1]
     URI_COUNT_ODD(1.00)[1,57]
     MX_INVALID(0.50)[cached]
     GREYLIST(0.00)[pass,body]
     MIME_GOOD(-0.10)[multipart/alternative,text/plain]
     NEURAL_SPAM(0.00)[0.302,0]
     IP_SCORE(-0.01)[ip: (0.52), ipnet: 192.92.97.0/24(0.35), asn: 32475(-0.08), country: US(-0.84)]
     TO_DN_ALL(0.00)[]
     TO_MATCH_ENVRCPT_ALL(0.00)[]
X-Rspamd-Server: nethservice.nethesis.it

nethbot commented 6 years ago

in 7.4.1708/testing:

nethbot commented 6 years ago

in 7.4.1708/testing:

stephdl commented 6 years ago

one way to get rid of actions in the rspamd WebUI is to forbid the /rspamd/actions and /rspamd/saveactions at the apache proxypass level, kudo to @DavidePrincipi :D

two workable propositions

# This is action view
<Location ~ ^/rspamd/actions($|/)>
    ProxyPass "!"
</Location>

# this is the save action
<Location ~ ^/rspamd/saveactions($|/)>
    ProxyPass "!"
</Location>

or

<Location ~ ^/rspamd/saveactions($|/)>
   RewriteEngine On 
   RewriteRule .* - [L,R=404]
</Location>

<Location ~ ^/rspamd/actions($|/)>
   RewriteEngine On
   RewriteRule .* - [L,R=404]
</Location>

I would go to the first proposal, for both the action score are no more displayed and symbols are still saved in the rspamd_dynamic map. In short I think we reach the goal, we still get the hand on settings in nethgui

DavidePrincipi commented 6 years ago

I read somewhere in rspamd config that rspamd_dynamic has priority 5, whilst override.d has priority 10. We could bypass UI settings with it...

stephdl commented 6 years ago

I believe it finished this honey time davidep, I wonder the documentation has not been updated, now the UI gets the precedence. I tested it again this wk: set a spam reject score in override configuration file, then increase it in the webUI, you will find that the UI reject score is used....please test it again, shout if i'm wrong

you read it at https://rspamd.com/doc/faq.html#where-does-the-webui-store-settings

nethbot commented 6 years ago

in 7.4.1708/testing:

nethbot commented 6 years ago

in 7.4.1708/testing:

nethbot commented 6 years ago

in 7.4.1708/testing:

nethserver-lib-2.2.6-1.2.g74675ad.ns7.noarch.rpm

nethbot commented 6 years ago

in 7.4.1708/testing:

nethbot commented 6 years ago

in 7.4.1708/testing:

nethbot commented 6 years ago

in 7.4.1708/testing:

nethbot commented 6 years ago

in 7.4.1708/testing:

DavidePrincipi commented 6 years ago

In nethserver-testing:

rspamd-1.7.3-1.x86_64.rpm
rspamd-debuginfo-1.7.3-1.x86_64.rpm

nethbot commented 6 years ago

in 7.4.1708/testing:

gsanchietti commented 6 years ago

Everything seems to work as expected: inside rpsmad UI the "Actions" section doesn't contain any control form.

nethbot commented 6 years ago

in 7.4.1708/testing:

nethbot commented 6 years ago

in 7.4.1708/testing:

nethbot commented 6 years ago

in 7.4.1708/updates:

DavidePrincipi commented 6 years ago

In nethserver-updates: nethserver-lib-2.2.7-1.ns7.noarch.rpm

NethServer / dev

Upgrade rspamd to 1.7.3 #5437