Using `/srv/<domains/mailboxes/<user>/forward` with an external destinations fails SPF and DKIM checks

patch0 commented 7 years ago

As /srv/<domains/mailboxes/<user>/forward simply relays the mail, it often causes SPF and DKIM signed mail to fail. This is particularly bad as the more important the mail (banks etc) the more tight their SPK/DKIM will be, and the destination mail server may simply drop invalid mail, or in many cases (see SNDS) simply flag all mail coming in that route as spam.

This can be worked around by using a basic Sieve filter (created in a mail client or Roundcube, etc) instead of a basic forward, which generates a header in the final mailbox like this:

Received-SPF: neutral (google.com: 2001:41c8:51:7a3::163 is neither permitted nor denied by best guess record for domain of admin@carth.ebonhawk.example.uk0.bigv.io) client-ip=2001:41c8:51:7a3::163;
To: nobody@under100words.com
From: Paul <paul@example.co.uk>
Subject: Test forward
Organization: Bytemark Hosting
Message-ID: <bdf891dd-fb0c-f7ca-3f89-7d7a00ff49ee@bytemark.co.uk>
Date: Fri, 24 Mar 2017 09:25:44 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="1kPfKMM3kgqtkMjBuqBIR3uSp4MpjjNV9"
Sender: Symbiosis Administrator <admin@carth.ebonhawk.example.uk0.bigv.io>

This works with a simple sieve rule like:

# rule:[Redirect all mail.]
if true
{
  redirect "under100words@example.net";
  stop;
}

It would be great if the forward file generated the relevant sieve rule, and massively improve deliverability.

Originally reported on Bytemark's Gitlab by @pcammish on 2017-03-24T09:53:18.430Z

patch0 commented 7 years ago

How is this different?

Originally posted by @patch0 on 2017-03-24T11:28:38.633Z

patch0 commented 7 years ago

Heres a header from a mail which was forwarded with the forward file (though a different domain, but on the same box).

Delivered-To: example.com@example.net
Received: by 10.157.19.41 with SMTP id f38csp10034ote;
        Fri, 24 Mar 2017 05:14:16 -0700 (PDT)
X-Received: by 10.28.216.141 with SMTP id p135mr2813314wmg.71.1490357656670;
        Fri, 24 Mar 2017 05:14:16 -0700 (PDT)
Return-Path: <paul.cammish@bytemark.co.uk>
Received: from carth.ebonhawk.example.uk0.bigv.io (under100words.com. [2001:41c8:51:7a3::163])
        by mx.google.com with ESMTPS id o51si3018032wrb.203.2017.03.24.05.14.16
        for <example.com@example.net>
        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
        Fri, 24 Mar 2017 05:14:16 -0700 (PDT)
Received-SPF: fail (google.com: domain of paul.cammish@bytemark.co.uk does not designate 2001:41c8:51:7a3::163 as permitted sender) client-ip=2001:41c8:51:7a3::163;
Authentication-Results: mx.google.com;
       dkim=pass header.i=@bytemark.co.uk;
       spf=fail (google.com: domain of paul.cammish@bytemark.co.uk does not designate 2001:41c8:51:7a3::163 as permitted sender) smtp.mailfrom=paul.cammish@bytemark.co.uk;
       dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytemark.co.uk
Received: from man.mx.bytemark.co.uk ([2001:41c8:20:862::184]) by carth.ebonhawk.example.uk0.bigv.io with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from <paul.cammish@bytemark.co.uk>) id 1crO6i-0000ka-4H for allmail@example.com; Fri, 24 Mar 2017 12:14:16 +0000
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bytemark.co.uk; s=20131115; h=Content-Type:MIME-Version:Date:Message-ID:Subject:From:To; bh=B5PkqhuntKWpysG+m8HT+RHrrDLsT/gjNyQiqzyNbrc=; b=eOq5NpNRJhDkGOQ2+f1mFzgseB2p1iwe6NamKpk9f2tGKdf1i6eDci//aLgqb0KvyWEFARb1kZgz0s+IolYF9DN6fXkYXWvX/ebIgUzcWPDrkDVvPfIyf1pbSDOJbPheulkc7noKeCpSSHEMZ35VQrBiOAup0LReOVcm6kfwTq9bDn0HQv6wU7TbpVlJZkqmtUb/eIMt/1/3ogbzOYJHIdEjhC4kC+FRvcAHu8jDqn4BYaHtqk32k4INcIRdCtEGQhglEy0Bsjfn9itNgvl3OH/2D5aghfdPdKEv+BKaXXcCv8i4jmTg6vtbAi52cbz1PXBF+SlalolbHEkMjNbHNA==;
Received: from ratatoskr.bytemark.co.uk ([2001:41c9:0:1017::48] helo=ratatoskr.dh.bytemark.co.uk) by man.mx.bytemark.co.uk with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from <paul.cammish@bytemark.co.uk>) id 1crO6h-00064F-U7 for allmail@example.com; Fri, 24 Mar 2017 12:14:15 +0000
Received: from [2001:41c8:3:1::186] by ratatoskr.dh.bytemark.co.uk with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from <paul.cammish@bytemark.co.uk>) id 1crO6h-0003Of-Qh for allmail@example.com; Fri, 24 Mar 2017 12:14:15 +0000
To: allmail@example.com
From: Paul <paul@bytemark.co.uk>
Subject: Non-SPF compatible forward
Organization: Bytemark Hosting
Message-ID: <d69b4edb-7596-996d-6433-ac720420b154@bytemark.co.uk>
Date: Fri, 24 Mar 2017 12:14:15 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="45j6fTKi7HhTnlIE1tViRHtOmNWMvkVC4"

Effectively it's just dropping the mail back in the spool after changing the RCPT TO - this is fine for delivery, but a sender with SPF set strictly will reject it as the original domain (bytemark.co.uk in this case) doesn't consider the Symbiosis server (carth.ebonhawk.example.uk0.bigv.io) as valid.

We've had more than a few customers over the last year who have been using forwards like this for account which get a fair bit of spam then forward the mail on to a hotmail address or similar - this looks like the server is sending the spam itself (complete with invalid SPF and so on), so if left a while the IP reputation will drop with the destination host, and then that server will have trouble sending mail to any account on that host.

Originally posted by @pcammish on 2017-03-24T12:30:11.344Z

patch0 commented 7 years ago

So from what I can gather here, the important difference is that the Return-Path is re-written from

Return-Path: <paul@bytemark.co.uk>

to

Return-Path: <admin@carth.ebonhawk.example.uk0.bigv.io>

by the sieve forwarding, but not by the exim4 forwarding. I'd almost be tempted to suggest that the sieve rule is doing the wrong thing anyway.

The Return-Path is where bounces will go to if the recipient address is unavailable for some reason. I'd assume that if under100words@example.net bounced, then admin@carth.ebonhawk.example.uk0.bigv.io would not be particularly interested in the bounce, especially as that email address is not even the one that forwarded the email (nobody@example.com). The Exim forward file leaves the return-path as-is, meaning bounces go back to the original sender, which I think is the correct thing to do.

This is a really common problem with SPF, and is "solved" with SRS although this would be quite complex to sort out, I suspect!

Originally posted by @patch0 on 2017-04-20T14:18:52.361Z

patch0 commented 7 years ago

Assuming someone is reading/receiving the local admin emails, then they would probably want to know if they set a forward which isnt working any more so they can disable it and/or contact the user by other means to sort it out.

Since I started looking at SNDS, the majority of the reports which were causing problems (for us and other users) were down to blind forwarding on noisy addresses, and while it's still happening (although at a much lower rate) if Symbiosis is being responsible with mail it would probably need to either use SRS (which would be non trivial) or rewrite the Return-Path to something relevant to avoid it becoming a bigger problem later on - after all, any mail with SPF set up properly will not be delivered, and we've seen that with our own customers using forwarding which breaks the line of communication in RT.

Originally posted by @pcammish on 2017-04-20T14:58:18.266Z

patch0 commented 7 years ago

Hmm, if the server is DKIM signing outbound mail, then that should be OK. But SPF checks fail without rewriting of the return path: and that will massively hinder deliverability. Especially when the sender address is in a domain with a "-all" spf record like, erm, BYTEMARK.CO.UK

SRS isn't enough these days, though. Big recipients like YAHOO also check SPF for the FROM header address. That's a big mess. Forwarding is basically not reliable these days, and we should discourage it, it my view. Instead, we should steer people to using POP/IMAP fetch.

Originally posted by @ieiloart on 2017-04-20T15:11:30.234Z

patch0 commented 7 years ago

@pcammish sure, but I'd argue it is the person that sent the email that will want to know if the delivery has failed, not the local admin.

@ieiloart, isn't that a violation of the spec? I could be wrong...

Originally posted by @patch0 on 2017-04-20T15:27:34.340Z

patch0 commented 7 years ago

Yes, SPF isn't supposed to be used to inspect message headers at all. The problem is, that validating the return path is pretty meaningless, since the person reading the message never sees that. Anyway, Yahoo do that because someone stole all their users' address books. So they have a massive fraud problem.

It means that discussion type mailing lists are pretty much unusable, unless you rewrite From headers, or don't have any Yahoo users on the list. But lots of users don't have sigs, so when you rewrite the From header, nobody on the list knows who wrote what. So mailing list managers typically offer rewrite options to change "Bill Blogs \foo@gmail.com\" to something along the lines "Bill Blogs via the list \list@example.com\"

Originally posted by @ieiloart on 2017-04-20T15:35:44.409Z

patch0 commented 7 years ago

I'm going to close this issue for now, as I don't think it can be addressed reasonably at this time.

BytemarkHosting / symbiosis

Using `/srv/<domains/mailboxes/<user>/forward` with an external destinations fails SPF and DKIM checks #54