mailcow / mailcow-dockerized

mailcow: dockerized - 🐮 + 🐋 = 💕
https://mailcow.email
GNU General Public License v3.0
8.29k stars 1.13k forks source link

Outlook/Office365/Microsoft365 and junk mails #2851

Open patschi opened 4 years ago

patschi commented 4 years ago

This issue is more to document and make people aware experiencing similar issues. Discussion, experiences or any tips to come to a solution might be helpful for everyone.


Office 365 / Outlook is quite special when it comes to get mails from your own mailserver delivered to said providers. In a negative aspect, unfortunately.

The problem There are many reports from users having issues to get serious legit mails delivered to Microsofts' mailing service correctly, even with state-of-the-art non-blacklisted mailservers using latest techniques like DKIM, ARC and strict SPF and being part of their JMRP and SNDS program. In most cases delivered mails are always moved into Junk/Spam folder for absolutely no reason.

Important to notice: This is not limited to mailcow instances overall and is an ongoing issue since a quite long time.

If you have customers at Office 365 or even worse: Outlook.com you should tell them about this issue and migrate them to another service, as they will not be able to receive legit mail from clean mail servers. Business critical mail may never reach their mailbox. This is not the senders problem, this is a serious problem for the recipient and therefore the Microsoft customer. Microsoft seems not to be able to handle their spam filters and tries to mitigate this problem by blocking whole foreign networks.

Solution Unfortunately there is no solution available yet. Several users (including me) tried to get more information and support from Microsoft, but without any noticable improvement nor helpful reply. Apparently Microsoft has no interests at all that their users and companies - relying on Office 365 - gets legit mails of any relevance delivered.

Even analysing all headers on the Microsofts' end after delivering just gives you cryptic headers without any sort of explanation why their considered mails as spam. There are several docs around explaining a few details, but so far they are all either outdated or useless.

Following GitHub issues at Microsofts' docs repository are still pending since a longer period of time to hopefully get some more information:

  1. https://github.com/MicrosoftDocs/OfficeDocs-o365seccomp/issues/743
  2. https://github.com/MicrosoftDocs/OfficeDocs-o365seccomp/issues/442
  3. https://github.com/MicrosoftDocs/OfficeDocs-o365seccomp/issues/409

What can you do? Basically nothing. This might be a workaround.

However you are greatly welcomed to push mentioned GitHub threads above to make Microsoft more aware about this serious issues on their end. If you have more direct connections, use them.

This is going to be continuously updated...

Adorfer commented 4 years ago

Been there, done that, since 10+ years... If your are stuck in this limbo at M$ (office365/live.com etc) for no obvious rasons: get new IP/IPv4 addresses for your mailserver.

My take on this is:

Invite users never to flag spam in their M$-boxes which had been forwarded through your server, especially if it's coming from 3rd party mailingslists: It helps nobody and causes a lot of frustration (especially for your users at M$ services)

patschi commented 4 years ago

This is indeed frustrating, and so far the only provider causing that amount of issues.

In my case, this happened during an ongoing mail conversation. In my opinion that is ridiculous and shouldn't happen at all. I was communicating with my new employer - which are hosting their mail-stuff on Office 365: We wrote about 20 serious, legit mails in both ways.

At some point their mail systems suddenly dropped my mails in the junk folders for absolutely no reason. I mean, I was writing with many different mail addresses on their end, which were replying as well. I don't get it, why their systems believe that this must be spam or something harmful. During an ongoing mail conversation. Beside the fact that mails were (detectable) replies, and not entirely new mails.

You might imagine how much confusion this has caused in the end... This can't be in the sense of Microsofts' customers.

JustAB0x commented 4 years ago

My current employer uses Office365 and we have issues like this all the time with smaller companies we work with. The craziest thing is: if i try to see why the email gets flagged (mail flow report via the secruity and compliance center), i just get "Flagged as Spam: No further information available".

I myself don't have this problem. I use their SNDS which may help out a bit.

patschi commented 4 years ago

I use their SNDS which may help out a bit.

Did it? Did you ever received a notification from their JMRP/SNDS? I've been in their programs since basically ever, and never got any message. I'm already wondering if this works at all :D

marrco commented 4 years ago

Did you ever received a notification from their JMRP/SNDS?

yes to my SNDS registered abuse@mydomain, also a few weeks ago from staff (at) hotmail dot com w/ subJ "complaint about message from xx.xx.xx.xx" without text and containing as an attachment the original message that a Hotmail user manually marked as spam.

No links for followup nor to mark the issue as resolved,

Adorfer commented 4 years ago

an attachment the original message that a Hotmail user manually marked as spam.

this is what i stated above: Some M$-Endcustmer (hotmail.com, live.com) flags messages coming through your system as spam:-> YOUR SERVERs IP IS GOING TO THE GULAG, NO APPEALS POSSIBLE.

JustAB0x commented 4 years ago

I use their SNDS which may help out a bit.

Did it? Did you ever received a notification from their JMRP/SNDS? I've been in their programs since basically ever, and never got any message. I'm already wondering if this works at all :D

"may help " i'm not sure. You do get insights into your IP status etc. So you at least now if they block you because of your IP or the messages specifically.

SomeGeek commented 4 years ago

My personal experience is that this problem is mainly on the free tier: outlook.com, hotmail.com, etc. Had this kind of problem a while ago, my mails where not allowed at all (blacklisted, for no reason, new server/ip). However, mails to the Office 365 paid subscription (someone with a own domain) received my mails just fine.

After multiple emails asking to at least say why I was blocked, they decided to 'mitigate' my ip for no reason. I never knew exactly it was blocked.

My take on this (and this might be a unpopular opinion, but regardless) is...

Punishing people who run their own server.

By making this thing a PITA, they can subtle convince people that it's not a good idea to run their own server at all (which I totally don't agree with, the more decentralized, the better), and switch to their service instead. Off course, there are other mail companies with a large customer base (and they can't afford to block them), but hey, every customer bringing their mails to them (including the data) is one right?

I don't see another good reason why they would do this.

patschi commented 4 years ago

There might be a possible workaround for some people: Routing all mails to *.outlook.com over a different mailserver, e.g. AWS SES, Sendgrid, Mailjet, etc. Somehow sad that this is the only way of getting mails over to Microsoft in a reliable way.

For some unimaginable reason (sarcasm!) there's even a Stackoverflow question with a good reply how to configure this sort of workaround using postfix's check_recipient_mx_access: https://serverfault.com/questions/663418/relay-host-based-on-destination-mx-record/663611#663611 (This can't be configured through mailcow UI, must be done manually)

mkuron commented 4 years ago

@patschi, I‘ve been using transport maps to deliver to {outlook,hotmail,live}.{com,de} via sendgrid and that works nicely. Sendgrid has a free tier if you send less than 100 messages per day.

Perhaps we could implement check_recipient_mx_access in Mailcow UI similar to how we deal with transport maps and add the workaround to the documentation. It‘s disappointing that we need to resort to solutions like this, but Microsoft seems to even blacklist "good" IP addresses if they don’t deliver a certain minimum number of messages per month.

patschi commented 4 years ago

I‘ve been using transport maps to deliver [...]

Yes, that's possible too, however this doesn't work on custom domains or Office365-hosted domains, therefore I personally like the check_recipient_mx_access-way a bit more.

Perhaps we could implement check_recipient_mx_access in Mailcow UI similar to how we deal with transport maps

Yes, I also think that this is a good idea to offer users at least a workaround for this mess. I've already told @andryyy about the idea - it's up to him to decide if and how exactly this should be integrated like.

andryyy commented 4 years ago

How is authentication handled? How does it interfere with transport maps? It is not as easy as setting up a second map.

andryyy commented 4 years ago

Something like "asd.com FILTER smtp:bla" can also lead to funny behaviors when you bcc an address. It will probably end up duplicated.

SomeGeek commented 4 years ago

It will be hard to prove that they're doing this on purpose, but if we can, that would mean they're abusing their market power, right? That could lead to some interesting legal consequences...

At least that's what I was thinking...

patschi commented 4 years ago

So apparently Microsoft decides to ignore all the feedback, to kill the conversation and uses an comment* as an excuse to completely lock down the issue: https://github.com/MicrosoftDocs/OfficeDocs-o365seccomp/issues/409#issuecomment-526797089. That's just so ridiculous. Seriously.

I'm wondering where's their respect (according their Code of Conduct) for all affected users which are lucky enough not to host their mailservers at Microsoft?

Conversation continued here... Wondering how long this issue stays open.

*To be fair, the comment was a little bit harsh, but neither wrong nor a reason to lock the conversation IMHO.

Adorfer commented 4 years ago

To conclude: No basic change in microsofts policy to silently block mailservers since 2005: not reacting to tickets to unblock, not caring about servers not listed on any rbl, with valid spl, dkim etc, send sending 99% of the mail volume from organic users.

If a client does not like to selfhost mail, then they should at least move over to google (they host 3rd party domains too) and provide good spam filtering.

aixtreme84 commented 4 years ago

There might be a possible workaround for some people: Routing all mails to *.outlook.com over a different mailserver, e.g. AWS SES, Sendgrid, Mailjet, etc. Somehow sad that this is the only way of getting mails over to Microsoft in a reliable way.

For some unimaginable reason (sarcasm!) there's even a Stackoverflow question with a good reply how to configure this sort of workaround using postfix's check_recipient_mx_access: https://serverfault.com/questions/663418/relay-host-based-on-destination-mx-record/663611#663611 (This can't be configured through mailcow UI, must be done manually)

I want to use that workaround. is the following way the right way ?

So i have to add check_recipient_mx_access hash:/opt/postfix/conf/finickydestination at

/opt/mailcow-dockerized/data/conf/postfix/main.cf

I have to create a file /opt/mailcow-dockerized/data/conf/postfix/finickydestination with this line .outlook.com smtp:[some_smtp.example.com]

andryyy commented 4 years ago

It will not work. You need to setup default_transport to route through a local smtpd like default_transpor t = [127.0.0.1]:2255. In this new smtpd, you can use check_mx_bla and content_filter through smtp: back into the mailcow system.

A lot of functions will break in mailcow.

SomeGeek commented 4 years ago

And all of a sudden, they blocked my server again. In the middle of a legitimate mail conversation, the mails started to bounce back:

Please contact your Internet service provider since part of their network is on our block list (S3150).

But, why?!? Here we go, again...

Edit: Off course.

Our investigation has determined that the above IP(s) do not qualify for mitigation.

@andryyy: What functions will break, exactly? Can those be worked around? This is a big issue that will render a mail server useless, if you take into account how much people are using Microsoft's email service.

Ry3nlNaToR commented 4 years ago

I've been using check_recipient_mx_access on my Mailcow server for a long time to route Microsoft bound mail via a relay I haven't seen any major issue, am not 100% sure what functions might get broken using that method.

The functions I think that might get messed up or broken would be things like Sender-dependent transports, BCC/Recipient/Transport Maps,Outgoing TLS policy map overrides which I don't use any of them anyway.

mgnisia commented 4 years ago

I would be also interested in a small how to in order to fix this problem! :) @Ry3nlNaToR could you maybe help us?

Edit update: Solved my issue via the a transport map in the routing menu.

andryyy commented 4 years ago

No @SomeGeek, it does not render your server useless. If your IP reputation sucks, use Amazon SES or any other relay service for Outlook or complain at Outlook.

You can implement it and test all kind of routing in mailcow afterwards. A lot of special-case routing will break. If it does not, create a clean and tested PR. I will merge it.

This does not mean, that you are unable to relay to Outlook. It is not a bug in mailcow. It is just that Outlook chose the easy path. You can ask people on Outlook.com how they feel about having a bunch of valid mail in junk, because Outlook is unable to filter inbound and forces its senders to scan mail for them. Tell them to whitelist you and tell them why. Ask them how they feel about checking their junk each day. They probably use it like their inbox now.

@Ry3nlNaToR you are correct, these are the functions, that will break. If you don't use them, it is fine, I guess. But you probably understand, I cannot implement it like this.

Github is NOT a place to get support for mail delivery. Buy a relay service, if you cannot reach the amount of mail per day, to be trusted by Outlook. Or just keep mailing. Tell your rcpts why this is happening. They will probably miss a lot of mail from small businesses.

gromain commented 4 years ago

Just for reference, this is happening to me too, on my own properly validated domain. It used to work at the beginning, but doesn't anymore for some reason.

It looks like Microsoft is getting busy not properly managing spam and yet still blocking legitimate emails from SBEs...

Anyway, I'll implement the workaround today with sendgrid. Thanks for the tip!

gromain commented 4 years ago

Hello,

I've tried the proposed fix but couldn't get it to activate using check_recipient_mx_access as @Ry3nlNaToR . I've had to use the transport map and do it manually (and it's not working for some unknown reason).

I'm not sure postfix can use both the sql database and the file-based system. Or at least, I couldn't configure it properly.

For the sake of it, here are my steps, from the folder /opt/mailcow-dockerized:

To test the filter, I put a BCC gromain@domain.org to data/conf/postfix/outlook in place of the FILTER part, but I saw the filter does not get activated and the smtp server never gets used. I believe the username and password would never get used anyway (since the database query on a host being defined).

I believe (but haven't tested yet due to lack of time) that if the check_recipient_mx_access is added to master.cf it will not break existing filters: https://marc.info/?l=postfix-users&m=121926229929879&w=2 .

andryyy commented 4 years ago

check_recipient_mx_access does break a lot of things. I tried it for a day and gave up, as there were too many configurations that needed to be changed - including routings in sql tables etc.

It works, if you don't mind breaking some other things we allow users to manage in mailcow UI. If you don't mind, you can implement it in your cow. :) Otherwise, talk to Outlook users and tell them, why they are missing important mail.

gromain commented 4 years ago

@andryyy I've tried setting up transport map, and it doesn't work. And I'm sorry, but this attitude of "talk to Outlook users" doesn't help. In my case, the mail is not even delivered, it's straight up rejected. And Microsoft is not willing to help, neither me directly nor through the users. So.... kind of the same as talking to a wall.

And let's be realistic, most of Outlook users are not going to switch overnight to Mailcow or another selfhosted solution, even if I would love for them to do it. There is a reason they pay Outlook instead of self-hosting. In the meantime, I still need to find a solution to have my mails delivered.

So, check_recipient_mx_access is not working well (if at all). Are transport maps the way to go? Should the postfix container be restarted after adding a new transport map?

andryyy commented 4 years ago

If it is rejected, mail them and ask to be removed from their bl or switch your provider.

I am not talking about moving them to mailcow. I moved a lot of MS hosted companys to Gmail, because they asked me, why they fail to receive mail from their customers. Gmail is not as aggressive as Outlook.

Just use a sender dependent transport map and assign it to your domain.

Or, well, switch providers. You are probably using OVH?

gromain commented 4 years ago

This is the problem, they can't remove me from their blacklist because I'm on Microsoft internal blacklist of some sort, not their own. So even if they whitelist me, it still gets rejected by Microsoft before. And switching provider is not a long term solution if I have to do it again in 6 months.

My mail server is hosted by Scaleway.

mgnisia commented 4 years ago

@gromain did you try the routing option in the admin menu with sendgrid/mailgun/AWS SES & etc. of mailcow?

Mailcow

gromain commented 4 years ago

Yes, but for some reason it doesn't work. That's why I was asking if a restart of postfix was needed. I'll test more now and report.

patschi commented 4 years ago

As a non-expert for postfix: I'm more thinking about a small script which is triggered when sending a mail.

  1. Sending mail
  2. Script checks if MX is O365 (before mail is actually sent)
  3. Adds transport map to be sent through a different mail relay
  4. Sends the mail as usual, with transport map applied

However I don't know if it is possible to solve in a reliable way.

gromain commented 4 years ago

There must be something I don't understand with the transport map. If I put .domain.org in destination, only subdomains mails are properly routed, main domain are not. So you have to be careful to put both as such: .domain.org, domain.org. It works for now, with Sendgrid. I'm redirecting only domain to which I can't send, and on the way of setting up Amazon SES globally (I mean, it's really inexpensive if it takes the pain out).

Adorfer commented 4 years ago
2. Script checks if MX is O365 (before mail is actually sent)

may you share that? What do you use for that? a DNSBL-information lookup on the mx like "SenderBase"? (query.senderbase.org) or is there even dedicated DNSBL-type service to determine "the big mx providers" (Google, MS, UI, DTAG...)? Setting up a local ASN->mailhosting-provider lookup table will probably a quite tideous task... plus keeping it up to date.

And do you cache the entries somehow?

patschi commented 4 years ago

may you share that?

It was just an idea. I don't have such a setup as I'm not a postfix-expert, unfortunately. Lookup might be only once, as the domain is added to the "Transport Maps" anyway - so basically sort of cache. If it is even possible to set this up this way.

SomeGeek commented 4 years ago

In my case, the mail is not even delivered, it's straight up rejected. And Microsoft is not willing to help, neither me directly nor through the users. So.... kind of the same as talking to a wall.

Exactly. And Hosted O365 is a different thing from the outlook.{...}, hotmail.{...}, msn.{...} free tier. Different blacklists. If your mails are not delivered to the free tier, there is a chance you don't have any delivery problems with the paid tier and vice versa (based on own experience!). You can't convince anyone to move their free hotmail/outlook account to Gmail because one IT guy has problems delivering their mail to them... They will to you it's your problem. And as a matter of fact it's kind of true... to make it worse.

This problem won't be fixed by Microsoft and most likely not your ISP neither. The latter one because the options are just limited... The only way to combat this is by finding solutions and implementing them on our side, because Microsoft won't change it's behavior (they have proven they don't gave a penny, didn't they?). It's a a very nasty situation, but we can't change the giant... We need to beat it. Unfortunately.

And if we keep the 'move away from Microsoft' mentality (which is best imho), we keep the delivery problems and nothing changes.

So, a good solution to route mails to stubborn companies trying to make life hard for anyone who gives a crap about decentralization via a different path is the only way to 'work around' this problem in the near future.

Is it a good solution? No. Is there a better solution, apart from moving your mail again? No. Would there be a better (legal) solution in the long term, like some lawsuit about market power abuse? Maybe. But not on the short term. Is it a solution that could work for us? Yes.

If the mountain will not come to us, then we must go to the mountain...

andryyy commented 4 years ago

It is not "one IT guy".

Feel free to contribute the feature without breaking mailcow as it is. It is a bunch of work and a lot of thinks need testing. There are so many scenarios which will definitely be overlooked... a dirty implementation is actually not hard. If you need it: implement it. This is how it works in open source projects. :) OR: Pay someones time.

It is not like there is no solution to this as of today... 1) write more mails to Outlook or 2) use a domain-wide relay and 3) do not use super big ISPs. Try to relay over smaller ASNs. I highly doubt they will ever risk to give a Hetzner VPS a good reputation, when they cannot be sure the next customer is a spam boy.

Adorfer commented 4 years ago

Does anyone have some email-Account i could send to at @live.com/@hotmail.com for testing? (I lack any contact to a person i know of having an emails acount the free thier of MS.)

andryyy commented 4 years ago

andre.peters@outlook.com is m test address. Please tell me, when you sent a mail. I don't check this account every day.

gromain commented 4 years ago

@Adorfer Yes, you can test with gromsex at hotmail.com, let me know when you do it, I don't check it everyday either.

warnerbryce commented 4 years ago

Someone said that : https://github.com/MicrosoftDocs/OfficeDocs-o365seccomp/issues/211#issuecomment-523865562

I had the same problem as my legit emails were landing in junk folder and then I removed these headers:

X-Fbl
'List-Unsubscribe-Post': 'List-Unsubscribe=One-Click'
X-Complaints-To
List-Id
X-Mailer

How can i mod the headers with the Dockerized Mailcow system ?

andryyy commented 4 years ago

We don't add these headers.

warnerbryce commented 4 years ago

Ping.... Do we have a solution ? Does someone have a script to change transport rules when it's a Exchange 365 Mail server ?

andryyy commented 4 years ago

Did you even bother to read my last answer?

gromain commented 4 years ago

I used Amazon send service for a while and it worked fine. It's a solution for low level of usage I guess. But in the end, I just had to let go Mailcow for a reliable hosted service I pay for. I had a bunch of issues with dropped received mail and in the end I did not trust mailcow anymore. At least my service is now reliable and I'm not banned for no reasons. And it's less expensive than hosting for a mailcow instance for my small org.

On Wed, 11 Dec 2019, 18:43 André Peters, notifications@github.com wrote:

Did you even bother to read my last answer?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mailcow/mailcow-dockerized/issues/2851?email_source=notifications&email_token=AAQWMVIKNO7F25UYIAFSWJDQYEREBA5CNFSM4IK4U4FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGT7KNI#issuecomment-564655413, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQWMVLWGEU2FNVZPIPN6FTQYEREBANCNFSM4IK4U4FA .

warnerbryce commented 4 years ago

Yes i saw it thank you. I wasn't speaking about the headers. I spoke about having a conf that send via a specific SMTP server everytime that you have an email address located on Microsoft servers (Outlook.com and Exchange 365). i have 10/10 with mail-tester.com. I don't use IPv6, i have SPF, rDNS, DKIM, DMARC and i am on the JMRP and SNDS of Microsoft that does nothing. When i use another SMTP server for outlook.com the mail goes in INBOX even if the SPF is bad ! It's really frustrating how Microsoft treats us, i look on the different issues on Github, it's unbelievable.

andryyy commented 4 years ago

Perhaps you should have bought a support package if you had trouble handling mailcow. ;)

There are a hundred reasons to not use a big hosted service (I say this providing such a service myself!).

If you get banned for no reason, you are probably on a shitty ASN, a range with a very bad, old reputation or just sent crap. I don't think it was you sending crap, but that's a valid reason for this to happen.

I provide a relay service for most customers, when I know their service is not sending bad mail. :)

So, no, I absolutely don't share your opinion but accept it. ;)

Edit: And yes, I know it sucks what MS is doing. But why would you play their game? Inform your recipients that they are filtering valid mail. I did this a lot of times and many reported, that they felt their filter is acting strange. Like every other invoice from Amazon is junk for MS etc.

I have many customers who care about their sensible data and won't ever use a shared service. Don't know. It probably depends on your needs on what you accept happens with your data. Hm.

gromain commented 4 years ago

If I could afford 30e per month, I would have. And i was with one of the biggest hosting providers in France. Apart from Microsoft shitty policies and Mailcow memory usage, it was fine!

On Wed, 11 Dec 2019, 20:17 André Peters, notifications@github.com wrote:

Perhaps you should have bought a support package if you had trouble handling mailcow. ;)

There are a hundred reasons to not use a big hosted service (I say this providing such a service myself!).

If you get banned for no reason, you are probably on a shitty ASN, a range with a very bad, old reputation or just sent crap. I don't think it was you sending crap, but that's a valid reason for this to happen.

I provide a relay service for most customers, when I know their service is not sending bad mail. :)

So, no, I absolutely don't share your opinion but accept it. ;)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mailcow/mailcow-dockerized/issues/2851?email_source=notifications&email_token=AAQWMVMYVA5QUZINLOU6EWTQYE4DJA5CNFSM4IK4U4FKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGUIEJA#issuecomment-564691492, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAQWMVOF63FCUS2V75XHLUTQYE4DJANCNFSM4IK4U4FA .

andryyy commented 4 years ago

OVH is one of the most blocked ASN. It is used to send spam a lot. Cannot really blame them, they are huge. :) I stopped using them for mailing without a relay.

warnerbryce commented 4 years ago

I might have a hint for having a Transport Map that send emails via Relay SMTP if the MX of the RCPT is on Microsoft' servers.

You can add a table with check_recipient_mx_access that let postfix search for the MX of the RCPT domain. But i don't know how to use this in paring with transport_map.

Does someone have a clue how to use it correctly ? http://www.postfix.org/postconf.5.html

patschi commented 4 years ago

Check the posts from back in October: It's not that easy as it may sound. See https://github.com/mailcow/mailcow-dockerized/issues/2851#issuecomment-542651667.