Emails going to spam because of improper header anonimization

UnleashedHost commented 5 years ago

Hello.

I have set up MailCow according to the official set up docs. My issue is that even though I have proper rDNS, SFP, DKIM and DMARC set up with all green checkmarks in the DNS test on MailCow them emails go to spam. Our domain isn't blacklisted on any of the lists.

The only issue I see there is that the emails are being sent out with multiple "Received from" headers. This is my whole message:

Received-SPF: pass (google.com: domain of nick@accuratenode.com designates 116.202.16.225 as permitted sender) client-ip=116.202.16.225;
Content-Type: multipart/alternative; boundary="----=_=-_OpenGroupware_org_NGMime-63-1548090141.974293-6------"
To: respect.comrad.ot@gmail.com, rozman.nik@gmail.com
MIME-Version: 1.0
Date: Mon, 21 Jan 2019 18:02:21 +0100
Subject: test
Message-ID: <3f-5c45fb00-b-534e2900@170800349>
From: "Nick | AccurateNode" <nick@accuratenode.com>

------=_=-_OpenGroupware_org_NGMime-63-1548090141.974293-6------
Content-Type: text/plain; charset=utf-8
Content-Length: 5

test

------=_=-_OpenGroupware_org_NGMime-63-1548090141.974293-6------
Content-Type: text/html; charset=utf-8
Content-Length: 17

<html>test</html>

------=_=-_OpenGroupware_org_NGMime-63-1548090141.974293-6--------

I have also tried configuring PostFix to use my custom config after trying your fix from another thread.

The link to the official one that I tried is: click me

I have additionally restarted those instances after each change and made sure I updated the containers. This seems to be occuring on the latest versions of it though as another install on another identical machine on the same network (different IP of course) works just fine. That one however was deployed 3 months ago.

Both servers are running on their own networks (same datacenter, different rack) the only difference between them is that one is running on Ubuntu 16.04.1 LTS and the other one on Debian 9.

Other than that I have tried editing the /etc/hosts file to see if that changed anything at all, but no change.

If you have a possible fix for this please let me know as soon as possible as we are unable to send out emails without them going to spam.

I am willing to provide credentials so you can see the situation for yourself.

Kind regards, Nick

Braintelligence commented 5 years ago

The thing is that specific mail hosters won't ever accept mails from fresh mail server IPs they've never seen before as anything but spam. This goes for example for microsofts mail servers.

Their users will have to proactively mark the mails as not-spam. At some point your mails will not be automatically flagged as spam then.

UnleashedHost commented 5 years ago

In my defense that same server has already been used to send out emails before with the same exact IP. We have reinstalled everything on it (even the OS) because we wanted to have a clean sheet. The reason for reinstallation was that we were switching to a different OS and because of a domain update.

Besides that the other server never sent to spam just minutes after its initial installation.

Braintelligence commented 5 years ago

Then maybe this is related: https://stackoverflow.com/questions/38420148/multiple-received-from-in-headers-while-sending-email-with-postfix

Multiple received from headers and also gmail.

UnleashedHost commented 5 years ago

Yeah. This is the exact one I've looked at before, but the issue isn't fixed.

Braintelligence commented 5 years ago

Let's wait for a member to look at this. If the multiple received-froms are the problem I sadly don't know which knob to turn to help you out ☹️.

UnleashedHost commented 5 years ago

Yeah. Thank you for your help anyways.

andryyy commented 5 years ago

Multiple Received headers are not a problem and quite normal.

Not being on a blacklist does not indicate you will land in any inbox. Outlook and Gmail hate new domains or old domains with new IPs. They will punish them 100 times and at some point accept them as ham.

116.202.16.225 seems to be a Hetzner IP. Their reputation isn't the best to begin with (Hetzner does an amazing job looking at their size!). Again: I don't mean blacklistings, your IP may never have seen a blacklist, it still belongs to a given ASN. Just like OVH. No matter the IP, you are doomed on 16276.

As long as you submit over your mail server and not directly communicate with Gmails MTA, you will have at least 2 hops (you -> your mta -> their mta). That's totally fine. More hops are not a problem, too.

UnleashedHost commented 5 years ago

I have another mail server running just a slightly older MailCow set up. Same scenario as this one. That one however doesn't send multiple headers. That one doesn't send the header on 127.0.0.1.

I've never had issues with that one even though it's on Hetzner as well. The IPs were updated on that one multiple times in a few months due to switching from one provider to another. I think I'll stick with Hetzner for now.

Any chance you could point me towards how to properly filter it or try to replicate it for yourself as I believe it is an actual bug and not just an issue that I might be having with it cause of the IPs.

andryyy commented 5 years ago

As long as you submit over port 587, you will always be the first hop, your MTA is the next hop, next/final is Gmail (for example) as "received by".

It is possible we replaced ALL headers previously, which was stupid. It does not change anything about the fact, that multiple received from headers are normal and don't get you into spam.

The 127.0.0.1 "received from" header does not indicate spam or something. Google does not punish that. This is a common first hop.

andryyy commented 5 years ago

Triggering @mkuron - you see a problem here?

mkuron commented 5 years ago

The multiple Received headers definitely isn't a problem. DKIM etc. validates correctly, so there is nothing you can do.

In my experience, Gmail isn't too bad in terms of deliverability -- unlike Microsoft, they don't just block everyone they don't know.

I suspect you just need to wait a bit until Google picks up that you aren't malicious. Send actual messages to Gmail users and have them manually move them out of spam. Sending messages containing just "test" doesn't help there because the filter looks at the message contents too. Also, check Google Postmaster Tools to see if that reports any specific problem.

UnleashedHost commented 5 years ago

Hello.

Thank you for replying. I am enforcing TLS on all in/out mail so I don't think that would be an issue. I'll try to do that. I have already registered on postmaster.google.com (i think its the correct domain?) and there is no indication of anything being wrong there.

I'll try to take into consideration your suggestion. Surely I can just plop Mailcow on some other server and it should work just fine or is there anything else that I'd have to do GMail wise like reporting an IP change or anything else that might limit me in the future ?

mkuron commented 5 years ago

Switch off TLS enforcement, just in case. Google supports TLS, but who knows what side effects it has. Also, check if you have any modifications in git: git diff origin/master. Try to keep your Mailcow as standard as possible while investigating this.

UnleashedHost commented 5 years ago

Alright. Will do.

UnleashedHost commented 5 years ago

This is the full diff:

diff --git a/data/conf/nginx/site.conf b/data/conf/nginx/site.conf
index 67d6f1a1..4ba05b19 100644
--- a/data/conf/nginx/site.conf
+++ b/data/conf/nginx/site.conf
@@ -17,6 +17,27 @@ map $sent_http_content_type $expires {
 }

 server {
+  listen 80;
+  listen [::]:80;
+  server_name autoconfig.*;
+  root /web;
+  location / {
+    fastcgi_split_path_info ^(.+\.php)(/.+)$;
+    fastcgi_pass phpfpm:9002;
+    include /etc/nginx/fastcgi_params;
+    fastcgi_param SCRIPT_FILENAME $document_root/autoconfig.php;
+    try_files /autoconfig.php =404;
+  }
+}
+server {
+  listen 80 default_server;
+  listen [::]:80 default_server;
+  include /etc/nginx/conf.d/server_name.active;
+  if ( $request_uri ~* "%0A|%0D" ) { return 403; }
+  return 301 https://$host$uri$is_args$args;
+}
+
+server {
   include /etc/nginx/mime.types;
   charset utf-8;
   override_charset on;
diff --git a/data/conf/postfix/anonymize_headers.pcre b/data/conf/postfix/anonymize_headers.pcre
index 099094d9..48cf4bdb 100644
--- a/data/conf/postfix/anonymize_headers.pcre
+++ b/data/conf/postfix/anonymize_headers.pcre
@@ -1,7 +1,6 @@
-if /^\s*Received:.*Authenticated sender.*\(Postcow\)/
 /^\s*Received:.*Authenticated sender:(.+)/
-  REPLACE Received: from localhost (localhost [127.0.0.1]) (Authenticated sender:$1
-endif
+    REPLACE Received: from localhost (localhost [127.0.0.1]) (Authenticated sender:$1
+/^\s*User-Agent/        IGNORE
 /^\s*X-Enigmail/        IGNORE
 /^\s*X-Mailer/          IGNORE
 /^\s*X-Originating-IP/  IGNORE
diff --git a/data/conf/postfix/main.cf b/data/conf/postfix/main.cf
index 83a252d8..6c649b27 100644
--- a/data/conf/postfix/main.cf
+++ b/data/conf/postfix/main.cf
@@ -132,7 +132,8 @@ smtp_sasl_password_maps = proxy:mysql:/opt/postfix/conf/sql/mysql_sasl_passwd_ma
 smtp_sasl_security_options =
 smtp_sasl_mechanism_filter = plain, login
 smtp_tls_policy_maps=proxy:mysql:/opt/postfix/conf/sql/mysql_tls_policy_override_maps.cf

mkuron commented 5 years ago

I‘m pretty sure there is more. Push the d key to scroll down in the diff.

-if /^\s*Received:.*Authenticated sender.*\(Postcow\)/

That‘s a bad idea as it breaks DKIM signatures on incoming emails. It‘s not what‘s causing your Gmail problem, but you‘ll want to revert this change.

Lajpat29 commented 5 years ago

Top mailbox providers(Google and Microsoft etc) blocks some of ASN based on their past spam history so, if even though ip is not on any blacklist, still email delivery fail due to such restrictions. I would suggest third party smtp relay like amazonses, mailgun and sendgrid etc for relay purpose as then you don't have to worry about ip blacklisting. But this workaround has drawbacks like loss of email privacy as now you are using third party server for relay. Mailgun provide option of using third party relay it can be easily configured from mailcow ui and then attach it to domain.

andryyy commented 5 years ago

That's right. IP blacklists are the worst thing to happen, but at least you can check against most of them. But you cannot check how some providers score your ASN. Large ISPs will also have their own DNSBL.

mkuron commented 5 years ago

You can also use Mailcow's transport maps (on the Routing tab) to only relay certain destinations via an external email provider. I use that to relay messages to outlook.com and hotmail.com via Sendgrid because Microsoft seems to effectively have a whitelist of IP addresses from which they accept messages. Sending to most other destinations should be fine as long as your server is on a reasonably reputable ASN.

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

mailcow / mailcow-dockerized

Emails going to spam because of improper header anonimization #2209