Rspamd doesn't respect mynetworks

kiriakos commented 3 months ago

Contribution guidelines

[X] I've read the contribution guidelines and wholeheartedly agree

I've found a bug and checked that ...

[X] ... I understand that not following the below instructions will result in immediate closure and/or deletion of my issue.
[X] ... I have understood that this bug report is dedicated for bugs, and not for support-related inquiries.
[X] ... I have understood that answers are voluntary and community-driven, and not commercial support.
[X] ... I have verified that my issue has not been already answered in the past. I also checked previous issues.

Description

It looks like the change to data/conf/rspamd/local.d/options.inc in PR #5812 introduces a behavior that softfails mail from additional hosts on the mynetworks property of postfix.

This can be circumvented by adding the hosts added to mynetworks also to the mailcow_networks.map but it isn't documented in Unauthenticated Relaying

Should this happen automatically or is the documentation missing the added step?

Logs:

postfix-mailcow-1    | Apr  5 06:13:11 1e8e9d6cb542 postfix/submission/smtpd[1431]: connect from unknown[10.1xx.xxx.xxx]
postfix-mailcow-1    | Apr  5 06:13:11 1e8e9d6cb542 postfix/submission/smtpd[1431]: discarding EHLO keywords: CHUNKING
postfix-mailcow-1    | Apr  5 06:13:11 1e8e9d6cb542 postfix/submission/smtpd[1431]: TLS SNI mailcow.redacted.redacted.com from unknown[10.1xx.xxx.xxx] not matched, using default chain
postfix-mailcow-1    | Apr  5 06:13:11 1e8e9d6cb542 postfix/submission/smtpd[1431]: Anonymous TLS connection established from unknown[10.1xx.xxx.xxx]: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256
postfix-mailcow-1    | Apr  5 06:13:11 1e8e9d6cb542 postfix/submission/smtpd[1431]: discarding EHLO keywords: CHUNKING
postfix-mailcow-1    | Apr  5 06:13:12 1e8e9d6cb542 postfix/submission/smtpd[1431]: 41A72281426: client=unknown[10.1xx.xxx.xxx]
postfix-mailcow-1    | Apr  5 06:13:12 1e8e9d6cb542 postfix/cleanup[1432]: 41A72281426: message-id=<>
rspamd-mailcow-1     | 2024-04-05 06:13:12 #46(normal) <3c3760>; task; rspamd_task_write_log: id: <undef>, qid: <41A72281426>, ip: 10.88.88.222, from: <root@sender.redacted.redacted.com>, (default: F (soft reject): [0.00/41.00] [ASN(0.00){asn:57381, ipnet:10.1xx.xxx.xxx/16, country:NO;},GREYLIST(0.00){greylisted;Fri, 05 Apr 2024 04:16:55 GMT;too early;}]), len: 244, time: 2.140ms, dns req: 1, digest: <7123b6474f0fee013d5a20dd35c5677b>, rcpts: <recipient@redacted.com>, mime_rcpts: <recipient@redacted.com>, forced: soft reject "Greylisted, please try again later"; score=nan (set by greylist)
rspamd-mailcow-1     | 2024-04-05 06:13:12 #46(normal) <3c3760>; task; rspamd_protocol_http_reply: regexp statistics: 0 pcre regexps scanned, 0 regexps matched, 3463 regexps total, 0 regexps cached, 0B scanned using pcre, 0B scanned total
postfix-mailcow-1    | Apr  5 06:13:12 1e8e9d6cb542 postfix/cleanup[1432]: 41A72281426: milter-reject: END-OF-MESSAGE from unknown[10.1xx.xxx.xxx]: 4.7.1 Greylisted, please try again later; from=<root@sender.redacted.redacted.com> to=<recipient@redacted.com> proto=ESMTP helo=<sender.redacted.redacted.com>
postfix-mailcow-1    | Apr  5 06:13:12 1e8e9d6cb542 postfix/submission/smtpd[1431]: disconnect from unknown[10.1xx.xxx.xxx] ehlo=2 starttls=1 mail=1 rcpt=1 data=0/1 commands=5/6

Steps to reproduce:

1. Add host to mynetworks
2. Start mailcow
3. Send mail from that host to mailcow: mail gets softfailed
4. Add host to `./data/conf/rspamd/custom/mailcow_networks.map`
5. restart rspamd
6. Send mail from that host to mailcow: mail gets delivered

Which branch are you using?

master

Which architecture are you using?

x86

Operating System:

Archlinux

Server/VM specifications:

16GB, 8 Core

Is Apparmor, SELinux or similar active?

no

Virtualization technology:

Xen

Docker version:

26.0.0

docker-compose version or docker compose version:

2.26.1

mailcow version:

2024-04-04_2309

Reverse proxy:

Nginx

Logs of git diff:

diff --git a/.gitignore b/.gitignore
index e25639a7..0e122d78 100644
--- a/.gitignore
+++ b/.gitignore
@@ -33,7 +33,6 @@ data/conf/portainer/
 data/conf/postfix/allow_mailcow_local.regexp
 data/conf/postfix/custom_postscreen_whitelist.cidr
 data/conf/postfix/custom_transport.pcre
-data/conf/postfix/extra.cf
 data/conf/postfix/sni.map
 data/conf/postfix/sni.map.db
 data/conf/postfix/sql

Logs of iptables -L -vn:

not relevant

Logs of ip6tables -L -vn:

not relevant

Logs of iptables -L -vn -t nat:

not relevant

Logs of ip6tables -L -vn -t nat:

not relevant

DNS check:

not relevant

FreddleSpl0it commented 3 months ago

Please try to add the host as Forwarding Host under System -> Configuration -> Options -> Forwarding Hosts

kiriakos commented 3 months ago

Please try to add the host as Forwarding Host under System -> Configuration -> Options -> Forwarding Hosts

Is this the nominal way? should this then be also added to the docs in Unauthenticated Relaying?

As the postfix part goes over the config files, it feels a bit unnatural to do the rspamd part over content (in the UI which presumably gets stored in a DB).

eakteam commented 3 months ago

The same issue happening to me... It worked fine before the latest update. After V.[2024-04] it fails. It doesn't respect mynetworks anymore at /opt/mailcow-dockerized/data/conf/postfix/extra.cf but when adding IPv4/IPv6 at System -> Configuration -> Options -> Forwarding Hosts it seems to send the mail NOT REJECTING but put it on SPAM FOLDER

@FreddleSpl0it Is this the right way to do that?

FingerlessGlov3s commented 3 months ago

Getting the same issue here as well, since updating. It's stopping some of my Linux postfix hosts from relaying emails.

I edited data/conf/rspamd/custom/ip_wl.map and added my IP range example 192.168.0.0/24, which has fixed the issue for now. Until a proper fix is done

gpz1100 commented 3 months ago

@FreddleSpl0it In my case, emails sent from a local unauthenticated server failed severely;

For now I've updated the ./data/conf/rspamd/custom/ip_wl.map as indicated above to include the permitted local subnets. However, this behavior was not present after the 2024-02 update and introduced recently. The past behavior still scanned the messages, which is fine and welcome, in case a server should go rogue.

Is this change a bug or there was specific reasoning behind it?

Edit: The other option is to comment out the line from the change (line 9 at https://github.com/mailcow/mailcow-dockerized/commit/75550eeea38badeb13d9198b9acedcbf132f3cb4#diff-8e0b9a448923c546d504adad891c8b76eccfe8024cb0aff1d2425845fb7d054dR9) . This seems to bring back the old behavior unless there's some side effects to be aware of?

FreddleSpl0it commented 3 months ago

This is not a bug. We got a report that outgoing spam filtering was skipped for every mail that was send unauthenticated from a private network. To fix this default behaviour we have set https://github.com/mailcow/mailcow-dockerized/pull/5812.

For everyone who is using unauthenticated relaying as described here https://docs.mailcow.email/manual-guides/Postfix/u_e-postfix-unauthenticated-relaying/, try to set local_addrs in data/conf/rspamd/local.d/options.inc to the same value as mynetworks in postfix extra.cf and then restart rspamd.

For example to enable unauthenticated relaying for 192.168.2.0/24

local_addrs = [127.0.0.0/8, ::ffff:127.0.0.0/104, ::1/128, fe80::/10, 172.22.1.0/24, fd4d:6169:6c63:6f77::/64, 192.168.2.0/24]

If this works i will update the documentation.

gpz1100 commented 3 months ago

@FreddleSpl0it Thank you for the explanation.

The above does indeed work. Is a ';' needed at the end of the line?

What happens to the /etc/rspamd/custom/mailcow_networks.map content?

Perhaps a more automated fix would be to update line 24 of https://github.com/mailcow/mailcow-dockerized/blob/master/data/Dockerfiles/rspamd/docker-entrypoint.sh to pass the contents of mynetworks from the postfix extra.cf file to mailcow_networks.map?

FreddleSpl0it commented 3 months ago

The ; is necessary for the configuration. The file /etc/rspamd/custom/mailcow_networks.map remains in the directory but will not be used for the local_addrs option anymore. This file contains mailcow's IPv4 and IPv6 networks. However, if you want to configure unauthenticated relaying, you'll need to manually set this in local_addrs as described in my example. The default IPv4 is 172.22.1.0/24 and IPv6 is fd4d:6169:6c63:6f77::/64.

We could consider specifying additional networks in mailcow.conf, rather than in extra.cf and options.inc to automate the process.

gpz1100 commented 3 months ago

My 2¢,

I know it will require some additional coding, but ultimately making that change in a single location (`mailcow.conf') would be a better solution in the long term and for troubleshooting.

Revisiting weeks/months later, unless one keeps good documentation, it will be long forgotten where the change(s) were made and will make support more challenging.

ShaunGibbins commented 3 months ago

would I need to move IPV6 config from data/conf/postfix/extra.cf to data/conf/rspamd/local.d/options.inc to disable IPv6 in postfix-mailcow?

From this doc https://docs.mailcow.email/post_installation/firststeps-disable_ipv6/

smtp_address_preference = ipv4 inet_protocols = ipv4

Or is /opt/mailcow-dockerized/data/conf/postfix/extra.cf still the place for this? Just got the impression that the config in extra.cf is now ignored?

gpz1100 commented 3 months ago

@ShaunGibbins With the modification of the local_addrs parameter in data/conf/rspamd/local.d/options.inc, all it's doing is telling rspamd not enforce spam checking on unauthenticated mail traffic originating from those networks. If you're not using ipv6 then there's no need to add any local ipv6 addresses to the line. I would however still include them to save yourself headaches in the future if you should device to use ipv6 on the mail server.

You're not moving anything from postfix/extra.cf to the rspamd..../options.inc even if you're not using ipv6. Just make the changes outlined in the doc you linked for disabling ipv6.

Recall, postfix is the MTA, while rspamd is for spam control.

Just got the impression that the config in extra.cf is now ignored?

Why are you getting this impression?

RedJohn14 commented 3 months ago

Hello everyone! After a long search I came across this entry and was able to solve my problem, great :-)

I have another e-mail system behind mailcow which sends outgoing e-mails via mailcow (DKIM is performed by mailcow here).

If I now "deactivate" the SPAM check with the entry and then send an EICAR test virus, for example, it is recognized by rspamd but not blocked. Is there any way I can counteract this so that outgoing e-mails are still checked for certain values?

Unless I list the IP in the local_addrs, rspamd always recognizes the emails as external, which I would like to avoid with the unauthenticated internal relay. At least I would like to reject / block outgoing SPAM and viruses.

Any ideas?

Rollbacke commented 2 months ago

Hi all, I was able to fix this by adding my LAN in System -> Configuration -> Options -> Forwarding Hosts On the other hand, I noticed that these emails were no longer signed by DKIM? (this is starting to cause problems with Gmail & co...) If I've an email from SOGo, everything is OK, the email has the DKIM signature. Anyone? EDIT: I believe that in fact the emails which do not come from the 172.22 network of mailcow were never signed. I don't believe this problem is related to this issue... I will add an custom MTA.

dragoangel commented 2 months ago

@Rollbacke answer is: https://github.com/mailcow/mailcow-dockerized/issues/5826#issuecomment-2041952576

Anybody here can do PR to docs to add missing parts. Automate via env var passing one line in 2 places sounds overkill, nobody will die if go in 2 configs instead of one. If every mailserver settings will be in mailcow.conf, you will get them into holy fire, nobody could manage it.

mailcow / mailcow-dockerized