search

forwardemail / forwardemail.net

Privacy-focused encrypted email for everyone. All-in-one alternative to Gmail + Mailchimp + Sendgrid.
https://forwardemail.net
Other
841 stars 110 forks source link

Messages Without DKIM Aren't Forwarded To Gmail #102

Closed hagai-helman closed 2 years ago

hagai-helman commented 3 years ago

Hello, and thanks for your great service!

It appears that some of the mail messages that are supposed to be forwarded to Gmail fail to arrive.

Here are the facts:

My bank sends me a mail notification for every transaction in my account. When it's sent directly to my Gmail address, it arrives without any problem (it passes SPF and DMARC, but doesn't have DKIM). When I configure my bank account to send the notification to my custom-domain address (which is supposed to be forwarded by ForwardEmail.net to the same Gmail address) I stop getting those messages all-together.

All SpamScanner features are disabled for this domain.

I tested what happens when I forward the messages to a Protonmail address instead of Gmail. They do arrive, but to the spam folder, and with the warning "This email has failed its domain's authentication requirements. It may be spoofed or improperly forwarded!" (Ironically, when the bank sends the notifications directly to the Protonmail address, they don't arrive at all).

For my specific use case, I found a workaround: I configured the bank account to send the notifications to a special address on my domain, which (using ForwardEmail.net) triggers a webhook. This webhook on my web server crafts a new message, and sends it to the "real" custom-domain address I want (and then ForwardEmail.net forwards it to my Gmail). Since I just want to know that a notification was received (and don't need to see the content), this is fine for me.

However, I suppose you may want to take a look at the problem, and find why it happens. I can send you (privately) examples of:

Please let me know if I can be of any assistance.

leftys commented 2 years ago

I encountered this issue as well. My understanding is that DMARC fails when one of SPF or DKIM is missing/failed. This is supposed to by solved by ARC protocol, but even though both forwardemail and gmail supports it, gmail doesn't seem to trust the ARC information. That's imho the root cause of problems. Therefore gmail's spam filter sees the messages as failing SPF and DMARC and either marks then as spam or even completely drops them(!).

I implemented a similar workaround with webhooks, storing messages and checking then once in a while not to miss any important email.

leftys commented 2 years ago

I found out some emails for me are also returned to their sender with DSN notification that the email was not delivered although they actually were. Together with the issue with disappearing emails and not logging DSNs/errors (https://github.com/forwardemail/forwardemail.net/issues/120), forwardemail is imho not suitable for use with gmail.

BrianJ20 commented 2 years ago

This exact problem is happening to emails received from the danish national digital ID system mitid.dk. The mails does not contain DKIM and after being forwarded via forwardemail gmail drops them completely.

I tried other email services and got the following results.

Forwardemail2Gmail Never Arrived Direct2Gmail Arrived Forwardemail2Outlook Arrived Direct2Outlook Arrived Forwardemail2ProtonMail Arrived marked as spam (This email has failed its domain's authentication requirements. It may be spoofed or improperly forwarded!) Direct2ProtonMail Never Arrived

onovy commented 2 years ago

I can confirm this issue. If sender domain has valid SPF, doesn't sign emails with DKIM and have DMARC records (p=reject), email is not delivered. DMARC specs allows this, it's valid.

DMARC record of sending domain:

_dmarc.domain.cz.   1800    IN  TXT "v=DMARC1; p=reject; adkim=r; aspf=r"

SPF record of sending domain:

domain.cz.      1800    IN  TXT "v=spf1 a -all"

Email is send from "A" record of domain.cz (SPF is valid).

Google's error:

Message failed: 550-5.7.26 Unauthenticated email from [domain.cz](http://domain.cz/) is not accepted due to domain's
550-5.7.26 DMARC policy. Please contact the administrator of [domain.cz](http://domain.cz/) domain if
550-5.7.26 this was a legitimate mail. Please visit
550-5.7.26  https://support.google.com/mail/answer/2451690 to learn about the
550 5.7.26 DMARC initiative. l7si35528240pfu.149 - gsmtp

I guess in this situation SRS should be used (to "fix" SPF).

onovy commented 2 years ago

Done more tests with same setup as above.

I can confirm that emails with valid SPF, without DKIM signature but with DMARC records p=reject are:

Related email headers after direct delivery to Gmail.

Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of test@domain.cz designates <ip> as permitted sender) smtp.mailfrom=test@domain.cz;
       dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=domain.cz

Related email headers after direct delivery to ProtonMail:

Authentication-Results: mailin031.protonmail.ch; dmarc=pass (p=reject dis=none)
 header.from=domain.cz
Authentication-Results: mailin031.protonmail.ch; spf=pass smtp.mailfrom=domain.cz
Authentication-Results: mailin031.protonmail.ch; arc=none smtp.remote-ip=<ip>
Authentication-Results: mailin031.protonmail.ch; dkim=none

Definitely error in ForwardEmail.net :/

niftylettuce commented 2 years ago

I think the reason is because it has passing ARC and the codebase right now only uses SRS if there was no ARC set.

Can you share if your test messages had ARC signature initially?

onovy commented 2 years ago

Nope, no ARC on sending side.

Headers on sending side (from my SMTP server):

Received: from __my_hostname__ (unknown [IPv6:__client_ipv6_ip__])
    by __smtp_hostname__ (Postfix) with ESMTPSA id 320484680055
    for <test@protonmail.com>; Fri, 18 Feb 2022 17:39:00 +0100 (CET)
Message-ID: <654352fe81cb7b68a44488b73e416258e0bf964c.camel@srcdomain.cz>
Subject: test without dkim2
From: test@srcdomain.cz
To: test@protonmail.com
Date: Fri, 18 Feb 2022 17:39:00 +0100
Content-Type: multipart/alternative; boundary="=-Zd56+W60piQcz/4F1E3W"
User-Agent: Evolution 3.36.5-0ubuntu1
MIME-Version: 1.0

Same email after received directly to ProtonMail:

Return-Path: <test@srcdomain.cz>
X-Original-To: test@protonmail.com
Delivered-To: test@protonmail.com
Received: from __smtp_hostname__ (__smtp_hostname__ [__my_ipv4__]) (using TLSv1.3 with
 cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
  key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No
 client certificate requested) by mailin015.protonmail.ch (Postfix) with ESMTPS id
 4K0cqy3HwVz7QQ5N for <test@protonmail.com>; Fri, 18 Feb 2022 16:40:14 +0000 (UTC)
Received: from __my_hostname__ (unknown [IPv6:__client_ipv6_ip__]) by
 __smtp_hostname__ (Postfix) with ESMTPSA id 320484680055 for <test@protonmail.com>;
 Fri, 18 Feb 2022 17:39:00 +0100 (CET)
Authentication-Results: mailin015.protonmail.ch; dmarc=pass (p=reject dis=none)
 header.from=srcdomain.cz
Authentication-Results: mailin015.protonmail.ch; spf=pass smtp.mailfrom=srcdomain.cz
Authentication-Results: mailin015.protonmail.ch; arc=none smtp.remote-ip=__my_ipv4__
Authentication-Results: mailin015.protonmail.ch; dkim=none
Message-Id: <654352fe81cb7b68a44488b73e416258e0bf964c.camel@srcdomain.cz>
Subject: test without dkim2
From: test@srcdomain.cz
To: test@protonmail.com
Date: Fri, 18 Feb 2022 17:39:00 +0100
Content-Type: text/html
User-Agent: Evolution 3.36.5-0ubuntu1
Mime-Version: 1.0
X-Pm-Spam: 0yeiAEic27iBOIJCpz91cGctFogIjgCwMINnjlJ3bjIgowgMClnJIc9Gy6ICdCItI
onovy commented 2 years ago

Email send thru same SMTP server (without DKIM signature but valid SPF) forwarded thru forwardemail.net to ProtonMail, found in spam folder:

Return-Path: <SRS0=1d3f=TC=srcdomain.cz=test@forwardemail.net>
X-Original-To: __final_address_after_forwarding@protonmail.com
Delivered-To: __final_address_after_forwarding@protonmail.com
Received: from mx1.forwardemail.net (mx1.forwardemail.net [138.197.213.185]) (using
 TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
  key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No
 client certificate requested) by mailin013.protonmail.ch (Postfix) with ESMTPS id
 4K0cqz6Bpkz7QQ65 for <__final_address_after_forwarding@protonmail.com>; Fri, 18 Feb 2022 16:40:15 +0000 (UTC)
Received: from localhost (localhost [127.0.0.1]) by __smtp_hostname__ (Postfix) with
 ESMTP id 64F8A4680185 for <__email_with_forwardemailnet_mx__@dstdomain.cz>; Fri, 18 Feb 2022 17:40:09 +0100 (CET)
Authentication-Results: mailin013.protonmail.ch; dmarc=fail (p=reject dis=none)
 header.from=srcdomain.cz
Authentication-Results: mailin013.protonmail.ch; spf=pass smtp.mailfrom=forwardemail.net
Authentication-Results: mailin013.protonmail.ch; arc=pass smtp.remote-ip=138.197.213.185
 arc.chain=:forwardemail.net
Authentication-Results: mailin013.protonmail.ch; dkim=none
Authentication-Results: mx1.forwardemail.net; dkim=none (message not signed); spf=pass
 (mx1.forwardemail.net: domain of test@srcdomain.cz designates __my_ipv4__ as
 permitted sender) smtp.mailfrom=test@srcdomain.cz smtp.helo=__smtp_hostname__;
 dmarc=pass (p=REJECT arc=none) header.from=srcdomain.cz header.d=srcdomain.cz;
 bimi=fail (missing bimi version in dns record) header.selector=default
 header.d=srcdomain.cz
Arc-Seal: i=1; a=rsa-sha256; t=1645202414; cv=none; d=forwardemail.net; s=default;
 b=kC+BwMwvymXAkyry9aNle5QZTT4K0LLkyr8HMyThy5bzi/kvfqBs8OuuDQ9hDcFjho6BGgYWq
 fDhY4dul8K4QK3m5SHbAMucMOJs6L+vZeuqBxPFHiLGpYVTlTa0XD9IlEtvZR1r8gsrSAIZBPfo
 85IbQYY7c8X67G3An3SttxM=
Arc-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=forwardemail.net;
 h=MIME-Version: Content-Type: Date: Message-Id: From: To: Subject; q=dns/txt; s=default;
 t=1645202414; bh=fFurW0LHr1i5ysHKC1wKhbEma4BU5B+L2migtG59vyM=;
 b=Nc0CzQ87WHsD0g4Y+hBkxaKBTmnfdyeg0te+hUY3vh8+BM71K5ckKWmYW7XxTAf6fgmfQqHKA
 BFfbtrpIuncM3+F19DvYz8NwFFPPL6xI/JUhl71k0pYTI5mJ7Gf5HF3y7L0Dq+GfQXwlRLWRa9B
 32qaq1yYTWiO1kYutXlCvBc=
Arc-Authentication-Results: i=1; mx1.forwardemail.net; dkim=none (message not signed);
 spf=pass (mx1.forwardemail.net: domain of test@srcdomain.cz designates
 __my_ipv4__ as permitted sender) smtp.mailfrom=test@srcdomain.cz
 smtp.helo=__smtp_hostname__; dmarc=pass (p=REJECT arc=none) header.from=srcdomain.cz
 header.d=srcdomain.cz; bimi=fail (missing bimi version in dns record)
 header.selector=default header.d=srcdomain.cz
Received-Spf: pass (mx1.forwardemail.net: domain of test@srcdomain.cz designates
 __my_ipv4__ as permitted sender) client-ip=__my_ipv4__;
X-Forwardemail-Sender: rfc822; test@srcdomain.cz
X-Forwardemail-Session-Id: gvrcspocaujb2f3i
X-Forwardemail-Version: 8.3.0
Subject: test without dkim3
To: __email_with_forwardemailnet_mx__@dstdomain.cz
From: test@srcdomain.cz
Message-Id: <1645202409420999427.6956145813939566831@test>
Date: Fri, 18 Feb 2022 17:40:09 +0100
Content-Type: text/html
Mime-Version: 1.0
X-Pm-Spam: 0yeiAEic27iBOIJCpz91cGctFogIjgCwMINnjlJ3bjIgowgMilnJIc9Gy6ICdCIxI
 AgLjZkFQRNVfgATNkQE9ogWT5mFQZMXghB3cSbwB9icmpWJYblG0pBSeyc0AB0MC2yAbMVCcbBib
 2cvNU6cmuDAINATwc1FMibwAIgLjBEVSRVESSZ0X0Tf1lGRESkVRR5UUPR0XUTJFMgTlvnJRbBSh
 gQmbWR25xvZWGGVcc9mtuJDICZsBZlZXuFxbI1GhgwWaGZt9luYWhyBccUmgmlGZmZyV50ZWtG4X
 M4CwQNFIlRQ9NTQVQFNIRojguV2cGZyV1hIGoGNdZMXgGB1UHIlJ9yY2uFxZIADuTBCMEUfZVMSE
 O19TT50FQNFIjRgoVMSEkyBTbV2zv5GICdwBJsdWoXNaIFGuQNFIiRSBNvZWcmRcbAiwgAjL0VJV
 RfUkSE9UVBCV6kkUFIzVMgZXum9bLNX0k5WYXYkJBvIHgnQcbVntyVmYGIvZBIciQFRVX4Ggw4CM
 EIUhxfTUTUVTUF0HCBSR0TZRBIOiME1VIlGu1x2YGZkVluIGlG1IcN3hcV2ZibwAggLjBVBTURlf
 UxUQ0XJRYgRkEk9QWoTgNRFSCThBQgbm4GVddBCw0JXYychBUgcmmGlZZVmy05WZGXg44xMCJE1I
 TVUfNRFSFTP9xZTkPEJIRkF6l1EI3chNUgZ2s25beBSogMXYGd4VwvdFtHRabBCNF1USHIhBRzcn
 tG4XM4CwfRFI0UDNJPX0fFlRVVEYM9FVUSF55vIElGRIcN2y0BXaWau9F2IGsWlYYJWsc5SZibwA
 AgLjNUlTR9VIM1EV0XO9lfTFMVVTVkEgsVXTGdwlJ0YXlG1IcN3hgU2Z2bs5BoeSgXMYdVG4vwFd
 HatRBNbCFU1SX4GgyFGcHdcNIgbigSwfINnygojIyeiAN0YWuW9aIojg1RmImbv5wgIism1IXN3j
 lJ3bjIgoAieyPFJURIC6y0CISNgwBSIlf0RTUJFFDlEREVPlI6TisDAIIJCBFRlQ1UyQogIjwTEL
 LACiUJUQVRUN9QMlEkVUSNUUO9USjIgowgMCBlNIIojgx4iMDMwAAgMDgSwfINnyiQWaiOiAIwNj
 imNZZEj11UDNGMjNNlNzhWRMNUz39BiIIH0=
X-Pm-Origin: external
X-Pm-Transfer-Encryption: TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
X-Pm-Content-Encryption: on-delivery
X-Pm-Spamscore: 101
X-Pm-Spam-Action: dunno
niftylettuce commented 2 years ago

I'll investigate this further and try to get back to you.

niftylettuce commented 2 years ago

I'm waiting on response from Google to see if we have been verified + added to their trusted list of ARC signers. If I don't hear back soon I'm going to do the Friendly From Rewrite for this edge case.

SteveDinn commented 2 years ago

I arrived here today because I have just noticed that I've been missing some emails that I should have been getting. I can't send an email from a gmail account to my forwarded domain (which is forwarded to another gmail address). It doesn't get sent to spam, it is just completely dropped. Hoping to find a solution to this!

hagai-helman commented 2 years ago

@SteveDinn

I can't send an email from a gmail account to my forwarded domain

Even though it's the same behavior, it sounds to me like it's a different bug, since messages from Gmail do have DKIM (the address the message is sent from ends with @gmail.com, right?).

I think you should do some experiments. Create another mail address (I recommend ProtonMail - it's easy to register, and like Gmail, provides a "View Headers" feature that helps debugging). Send a message from there to your forwarded domain, and see if it arrives. Then add the new address to your domain settings at ForwardEmail, so messages will be forwarded to it, and send a message from your Gmail address. Finally, send another message from the new address. The outcomes of these actions may give some useful hints about your situation.

SteveDinn commented 2 years ago

@hagai-helman Yeah, it's probably not the same issue. I didn't do anything, but I started receiving the emails about 18 hours after they were sent. They're still trickling in. It's more like they were all delayed in a queue rather than backlogged and sent all at once.

onovy commented 2 years ago

I'm waiting on response from Google to see if we have been verified + added to their trusted list of ARC signers. If I don't hear back soon I'm going to do the Friendly From Rewrite for this edge case.

did Google replied?

titanism commented 2 years ago

This issue is now resolved.