mailcow / mailcow-dockerized

mailcow: dockerized - ๐Ÿฎ + ๐Ÿ‹ = ๐Ÿ’•
https://mailcow.email
GNU General Public License v3.0
8.36k stars 1.13k forks source link

Outlook/Office365/Microsoft365 and junk mails #2851

Open patschi opened 4 years ago

patschi commented 4 years ago

This issue is more to document and make people aware experiencing similar issues. Discussion, experiences or any tips to come to a solution might be helpful for everyone.


Office 365 / Outlook is quite special when it comes to get mails from your own mailserver delivered to said providers. In a negative aspect, unfortunately.

The problem There are many reports from users having issues to get serious legit mails delivered to Microsofts' mailing service correctly, even with state-of-the-art non-blacklisted mailservers using latest techniques like DKIM, ARC and strict SPF and being part of their JMRP and SNDS program. In most cases delivered mails are always moved into Junk/Spam folder for absolutely no reason.

Important to notice: This is not limited to mailcow instances overall and is an ongoing issue since a quite long time.

If you have customers at Office 365 or even worse: Outlook.com you should tell them about this issue and migrate them to another service, as they will not be able to receive legit mail from clean mail servers. Business critical mail may never reach their mailbox. This is not the senders problem, this is a serious problem for the recipient and therefore the Microsoft customer. Microsoft seems not to be able to handle their spam filters and tries to mitigate this problem by blocking whole foreign networks.

Solution Unfortunately there is no solution available yet. Several users (including me) tried to get more information and support from Microsoft, but without any noticable improvement nor helpful reply. Apparently Microsoft has no interests at all that their users and companies - relying on Office 365 - gets legit mails of any relevance delivered.

Even analysing all headers on the Microsofts' end after delivering just gives you cryptic headers without any sort of explanation why their considered mails as spam. There are several docs around explaining a few details, but so far they are all either outdated or useless.

Following GitHub issues at Microsofts' docs repository are still pending since a longer period of time to hopefully get some more information:

  1. https://github.com/MicrosoftDocs/OfficeDocs-o365seccomp/issues/743
  2. https://github.com/MicrosoftDocs/OfficeDocs-o365seccomp/issues/442
  3. https://github.com/MicrosoftDocs/OfficeDocs-o365seccomp/issues/409

What can you do? Basically nothing. This might be a workaround.

However you are greatly welcomed to push mentioned GitHub threads above to make Microsoft more aware about this serious issues on their end. If you have more direct connections, use them.

This is going to be continuously updated...

admir86 commented 4 years ago

How I have to set next hop? What's the meaning of the color at username. If I change next hop to port 25 the bar becomes green.

Is that correct? image

Mails don't arrive, does I have to make some more configuration?

edit: I had to validate/confirm my sender domain at mailjet ๐Ÿ˜ฌ now it's working fine, mails are delivered in the inbox not junk mail

Lennix commented 4 years ago

Just wanted to jump into the discussion, since I'm also banned now. 24 months old IP, not blacklisted anywhere, using dkim and spf. I'm also registered at their stupid SDNS, which tells me "All of the specified IPs have normal status.".

And the only response from microsoft: "Our investigation has determined that the above IP(s) do not qualify for mitigation."

I will also talk to my customers that they should tell their outlook-contacts that they're unable to mail them. I am currently unwilling to waste my time because of that incompetent behemoth.

alexanderadam commented 4 years ago

check_recipient_mx_access does break a lot of things. I tried it for a day and gave up, as there were too many configurations that needed to be changed - including routings in sql tables etc.

It works, if you don't mind breaking some other things we allow users to manage in mailcow UI.

@andryyy / @patschi would it make sense to create another ticket for it (i.e. "_add support for check_recipient_mx_access_") where would be a checklist or list of tasks that shows which parts of mailcow will break at the moment (= what is still to be done) and which are already fixed?

Because I guess it will be easier if we could break it down into smaller tasks that can be more realistically be done by (potential) contributors.

andryyy commented 4 years ago

You can go ahead and analyze it. I did previously and stopped after too many critical changes.

Not working on this at the moment. If you think it works and does not break, go ahead and change it locally. :)

alexanderadam commented 4 years ago

So does this mean you have no interest in breaking the problem down or share your findings (what broke for which reason / what to look out for)?

andryyy commented 4 years ago

What?

How about you start working on it and contribute your work here?

alexanderadam commented 4 years ago

See, if you, as the author, even struggled with it, it is most probably even more difficult to solve this for someone who will also have to put some effort getting to know the code base.

That's why I wrote that it might be easier to contribute, if this issue (which is a complex one, as you stated earlier) could be broken down into smaller issues.

You stated that you invested one day already and I guess you got some insights of it (i.e. what you tried and what broke). If you could explain briefly what you tried and why it failed, the day you invested earlier, wouldn't be "lost".

LHozzan commented 4 years ago

I attempt to solve similar problems - we have many Postfix email servers, but sometimes we cannot delivery regular emails to M$ services. We can obtain complette email with all headers, but without information, what headers and values are what or what is exact reason, why specify emails was marked as spam or "suspiciously", we are only able to ask customers to ask his customers not use M$ email services or accept "lost emails" ... I fully understand, that "some bad guys" exist and can use this information against M$ services. This is just a M$ problem to improve their protection! So, if M$ want this way, OK, not use M$ services is choice too. ... and yes, I am very frustrated, because M$ ask for this services huge amount of money with zero effort to support third party!

SomeGeek commented 4 years ago

New problem: The mails to Outlook all get accepted:

XXXXXXXXXXX: to=<<email>@outlook.com>, relay=outlook-com.olc.protection.outlook.com[104.47.55.33]:25, delay=3.4, delays=0.61/0.03/1.4/1.4, dsn=2.6.0, status=sent (250 2.6.0 <xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@domain.tld> [InternalId=00000000000000, Hostname=MW2NAM10HT091.eop-nam10.prod.protection.outlook.com] 9770 bytes in 0.325, 29.270 KB/sec Queued mail for delivery -> 250 2.1.5)

But they never arrive. Not in the inbox, not in the spambox. Also, there is no bounce. So, looks like the new MO is using shadow banning rather than announcing the ban.

This is bad practice, as code 250 should indicate that the message should hit the mailbox. Also a very smart move, as they Microsoft can just ignore that there is a issue (hence the 250 status).

Edit: For those of you who are wondering, I am not the only one facing this problem... And there are lots of more topics covering the same issue. No solution has been found yet.

max-foss commented 4 years ago

I have a found solution that worked for me and is not listed here. First I had the problem that my e-mails were rejected because of "too many recipients" (less than 10, just informing a few people on O365 in a company of my new e-mail).

Registering my IPv4 at https://sendersupport.olc.protection.outlook.com/snds/JMRP.aspx (requires a free Microsoft account), only a single IPv4 works, everything else (range, CIDR) errors out with weird messages, did work fine for me, everything ends up in the respective inboxes. But I have to add, I have my own brand-new ASN and brand-new 100% clean IPv4-space.

andryyy commented 4 years ago

Hm, they told me it has no effect on deliverability, but you can and should monitor your IP space there.

A clean IP space and new ASN is ultimately the best solution. :-) You can gain a good reputation. A huge provider like OVH with a bad reputation in general (AS16276...) will truely never deliver to their inbox, I guess.

If anyone needs access to a free mail relay, write me to info@servercow.de - but no commercial usage and no business use, please. I also don't ask other companies to support me for free, I think that's fair. ๐Ÿ˜„

max-foss commented 4 years ago

Hm, they told me it has no effect on deliverability, but you can and should monitor your IP space there.

A clean IP space and new ASN is ultimately the best solution. :-) You can gain a good reputation. A huge provider like OVH with a bad reputation in general (AS16276...) will truely never deliver to their inbox, I guess.

If anyone needs access to a free mail relay, write me to info@servercow.de - but no commercial usage and no business use, please. I also don't ask other companies to support me for free, I think that's fair. ๐Ÿ˜„

E-Mail was rejected, tried sending it again, rejected, decided to add my IPv4 to my Microsoft account and specify how I want to get notifications of spam. After that I was able to successfully deliver this email from the queue and all further emails worked on the first try.

andryyy commented 4 years ago

This is not about being rejected but delivery to the inbox.

andryyy commented 4 years ago

image

๐Ÿ˜„ Best spam filter ever. All of this is flagged as non-junk.

Well done, Microsoft, well done.

jjkondrat commented 4 years ago

I had a similar problem with outlook.com/hotmail.com blocking emails to a few recipients since I host my mailcow domains on Amazon AWS.

I opened up a ticket with Microsoft, escalated to a real person, provided the info and my Static IP with evidence I was following all the expected practices and even though it took a few days... as expected they fixed the item. I would expect this process would work for anyone not sending spam....

Hello,

Thank you or contacting the Outlook.com Deliverability Support.

We have implemented mitigation on the IP ##.###.###.### and this process may take 24 - 48 hours to replicate completely throughout our system.

Sincerely, Outlook.com Deliverability Support

shaneonabike commented 4 years ago

Hey @jjkondrat this is great information and something I would be motivated to do. Where did you fill that out to report an issue / ticket? I think that's the part I most struggle with at the moment.

andryyy commented 4 years ago

That doesn't help. That's the default response you get, when they remove you from the blacklist. That's nothing they do to not mark your mail as spam.

The recipient needs to open the mail and select "mark sender as safe" from the options. Clicking "not junk" is not enough.

jjkondrat commented 4 years ago

Yeah, I misunderstood at some point.... if they're going to junk @andryyy is correct.... if they're not getting delivered at all and they were rejected based on my IP location is the problem i had.

andryyy commented 4 years ago

image

SomeGeek commented 4 years ago

Doesn't work when you have the problem described earlier: No error, the mail gets accepted (250) and then disappears (never arrives, not in the inbox, not as junk)...

EricThi commented 3 years ago

Hello,

For me, I have at this time fix all case on gmail and outlook/hotmail. for the last, between 1-2 week for fix it.

If can help people, my little way to fix junk to microsoft mail (and little on gmail)

1) configure reverse IP/spf/dmarc/dkim on dns side for your domain

example for me v=spf1 ip4:ip.of.you.server.mailcow include:_spf.google.com include:domain.of.your.server.smtp ~all

v=DMARC1;p=none;fo=1;adkim=r;aspf=r;pct=100;rf=afrf;ri=86400;sp=none;rua=mailto:your@email;ruf=mailto:your@email => not for microsoft, only google send all day a mail if my mail are pass on .zip with .xml on this mail <<

2) we need to register/check/test

to snds : https://sendersupport.olc.protection.outlook.com/snds/ => you can monitor report by microsoft network and get data on some days (important before contact him)

check your reputation, example : https://talosintelligence.com/reputation_center => on my case, i'm neutral (no mass mail send on internet :))

test connectivity on microsoft : https://testconnectivity.microsoft.com/tests/o365 (info : some tools don't work <<)

Check if your ip is on rbl or not (for google, i have clean on one rbl (a little, and old entry (end of 2018) + includ on spf and fix it) => check on many many sites :)

request to delist ip on microsoft : https://sender.office.com/

3) and after, if your mail are always move to junk :

go request to Outlook.com Deliverability Support via : https://sendersupport.olc.protection.outlook.com/pm/policies.aspx, you can found : Additional Resources

Deliverability Support
If you are adhering to the guidelines, practices and policies presented on this page and are still experiencing deliverability issues, please contact Outlook.com deliverability support. If you are not in compliance with the above policies and guidelines, it may not be possible for our support team to assist you.
    Outlook.com Deliverability Support

to send request here : https://support.microsoft.com/en-us/supportrequestform/8ad563e3-288e-2a61-8122-3ba03d6b8d75

=> here, you send a mail to human :) first reply (same day to send request !!!) 02 june 2020, last mail after check + mitigation + new delist (previous link) => all is good on 16 june 2020 ;)

4) Bonus :

After mail are good (not send "spam" before contact delivery support xD) for help microsoft/google to get data, i have create 4 little crontab on my server to send mail via my mailcow to gmail and outlook address :

@hourly cp /usr/local/smtp/template-joke_jod /tmp/joke_jod && curl -X GET "https://api.jokes.one/jod" -H  "accept: application/json" -H  "content-type: application/json" -H  "X-JokesOne-Api-Secret: api_key" | jq '.. | .text?' | sed 's/null//g' | sed 's/\\r/ /g' | sed 's/\\n/ /g' >> /tmp/joke_jod && cat /tmp/joke_jod | msmtp -a joke your.email@outlook.com, your.email@gmail.com && sleep 2 && rm /tmp/joke_jod

@hourly cp /usr/local/smtp/template-joke_blonde /tmp/joke_blonde && curl -X GET "https://api.jokes.one/jod?category=blonde" -H  "accept: application/json" -H  "content-type: application/json" -H  "X-JokesOne-Api-Secret: api_key" | jq '.. | .text?' | sed 's/null//g' | sed 's/\\r/ /g' | sed 's/\\n/ /g' >> /tmp/joke_blonde && cat /tmp/joke_blonde | msmtp -a joke your.email@outlook.com, your.email@gmail.com && sleep 2 && rm /tmp/joke_blonde

@hourly cp /usr/local/smtp/template-joke_animal /tmp/joke_animal && curl -X GET "https://api.jokes.one/jod?category=animal" -H  "accept: application/json" -H  "content-type: application/json" -H  "X-JokesOne-Api-Secret: api_key" | jq '.. | .text?' | sed 's/null//g' | sed 's/\\r/ /g' | sed 's/\\n/ /g' >> /tmp/joke_animal && cat /tmp/joke_animal | msmtp -a joke your.email@outlook.com, your.email@gmail.com && sleep 2 && rm /tmp/joke_animal 

#Baconipsum
*/5 * * * * cp /usr/local/smtp/template-baconipsum /tmp/baconipsum && wget "https://baconipsum.com/api/?type=meat-and-filler&paras=5&format=text" -O /tmp/baconipsumtmp && cat /tmp/baconipsumtmp >> /tmp/baconipsum && cat /tmp/baconipsum | msmtp -a joke your.email@outlook.com, your.email@gmail.com && sleep 2 && rm /tmp/baconipsum*

my "template" is just for beautiful mail : cat /usr/local/smtp/template-joke_jod

From: your.email@from.mailcow
To: your.email@outlook.com, your.email@gmail.com
Subject :

your.email@gmail/outlook.com = mail create by me for receive this beautiful mail => i read all mail via mark all mail has read and remove it once a week +/- :)

per day, i send 360 mail automatically to gmail and same to outlook

Bonusยฒ : You can see report on microsoft via snds and need a good reverse IP (number of mail with score and number of transaction on smtp with microsoft) (if we need to add range ip, is based on mail set on whois... if cannot change it you need to add all ip one by one :))

Screenshot_2020-09-04 SNDS - View Data

Example of report by dmarc by google : attachement .zip on mail with .xml : You can see count = number of mail delivered on gmail address :)

<?xml version="1.0" encoding="UTF-8" ?>
<feedback>
  <report_metadata>
    <org_name>google.com</org_name>
    <email>noreply-dmarc-support@google.com</email>
    <extra_contact_info>https://support.google.com/a/answer/2466580</extra_contact_info>
    <report_id>xxxx</report_id>
    <date_range>
      <begin>xxx</begin>
      <end>xxx</end>
    </date_range>
  </report_metadata>
  <policy_published>
    <domain>my.domain</domain>
    <adkim>r</adkim>
    <aspf>r</aspf>
    <p>quarantine</p>
    <sp>none</sp>
    <pct>100</pct>
  </policy_published>
  <record>
    <row>
      <source_ip>my.ip</source_ip>
      <count>361</count>
      <policy_evaluated>
        <disposition>none</disposition>
        <dkim>pass</dkim>
        <spf>pass</spf>
      </policy_evaluated>
    </row>
    <identifiers>
      <header_from>my.domain</header_from>
    </identifiers>
    <auth_results>
      <dkim>
        <domain>my.domain</domain>
        <result>pass</result>
        <selector>my.dkim.selector</selector>
      </dkim>
      <spf>
        <domain>my.domain</domain>
        <result>pass</result>
      </spf>
    </auth_results>
  </record>
</feedback>

edit : and i have 3 personal domains names on my mailcow => any error for these domains :)

Edit 30/12/20 : I have change today some params on my dns zone (clean) and reduce spf : "v=spf1 mx a -all" Why ? because on this website : https://www.dmarcanalyzer.com/fr/spf-4/checker/ => to many include, spf can be skip by isp and include google and cie is useless...

After, i validate my spf via : https://www.kitterman.com/spf/validate.html?

send mail to check spf/dkim : http://www.open-spf.org/Tools/

and found best practice for send mail to google and office with spf and dkim : https://emailtrends.com/news/2020-dmarc-works-autumn-update/

patschi commented 3 years ago

Relevant update mostly interesting for users located in Germany: Christian Huber, CTO von meerfarbigISP, hat eine Beschwerdeverfahren bei der Bundesnetzagentur in Deutschland hierzu eingereicht, siehe hierzu: https://twitter.com/glob3x/status/1318473060383686656

Erste Anbieter scheinen ebenfalls Interesse zu haben: https://twitter.com/IPProjects/status/1318566195197607942 https://twitter.com/netcup/status/1318812003243331584 https://twitter.com/phpfriends/status/1319014114979684353

Summary in English: The CTO of meerfarbigISP Christian Huber filed a complaint to Germany's regulatory office "Bundesnetzagentur" due to this unfair measures Microsoft has in place with Outlook and Office365. Some service providers appear to be interested joining the fight.

shiz0 commented 3 years ago

Now THIS is some GREAT news. Let's hope it'll improve the situation.

jol64 commented 3 years ago

I succeeded recently getting removed from the blacklist by filing a privacy complaint with Microsoft. A blacklist is an automated decision, and imho there has to be a means to appeal or otherwise Microsoft violates EU-GDPR Art 22. Not sure the colleagues at Microsoft understood the legal aspect, but at least they cooperated and I got removed now. Took me about 4 months. I definitely encourage others to do try the same. Best Regards, Joachim Disclaimer: I am not a lawyer, only a data protection fan... just expressing my own thoughts and not giving legal advice.

warnerbryce commented 3 years ago

@jol64 Hi, could you tell us where did you start to ask this request ? How did you do it ?

jol64 commented 3 years ago

the web changes frequently and looks different now. try starting here: https://www.microsoft.com/en-us/concern/privacyrequest-other

CookieCr2nk commented 3 years ago

Quoting https://github.com/mailcow/mailcow-dockerized/issues/2851#issuecomment-686356475

Hello,

For me, I have at this time fix all case on gmail and outlook/hotmail. for the last, between 1-2 week for fix it.

If can help people, my little way to fix junk to microsoft mail (and little on gmail)

1. configure reverse IP/spf/dmarc/dkim on dns side for your domain

[...]

Thanks This Helped for me ๐Ÿ‘ . Do you also see something in Google Postmaster? (https://www.gmail.com/postmaster/)

FingerlessGlov3s commented 3 years ago

I did step 3 from that guide but got the following reply from Microsoft.

Our investigation has determined that the above IP(s) do not qualify for mitigation.

Please ensure your emails comply with the Outlook.com policies, practices and guidelines found here: http://mail.live.com/mail/policies.aspx.

To have Deliverability Support investigate further, please reply to this email with a detailed description of the problem you are having, including specific error messages, and an agent will contact you.
Stealthbird97 commented 3 years ago

I did step 3 from that guide but got the following reply from Microsoft.

Our investigation has determined that the above IP(s) do not qualify for mitigation.

Please ensure your emails comply with the Outlook.com policies, practices and guidelines found here: http://mail.live.com/mail/policies.aspx.

To have Deliverability Support investigate further, please reply to this email with a detailed description of the problem you are having, including specific error messages, and an agent will contact you.

I got that. Explain the issue, explain that you are certain you are in compliance. Suggest that they prove that you are not in compliance and that the burden of proof is on them, by asking them to provide the "Spam" emails which resulted in you being on their blacklist and accuse them of monopolistic and anti-competition practices by pushing out small players etc. At least that worked for me, the second time it happened to me.

FingerlessGlov3s commented 3 years ago

I got that. Explain the issue, explain that you are certain you are in compliance. Suggest that they prove that you are not in compliance and that the burden of proof is on them, by asking them to provide the "Spam" emails which resulted in you being on their blacklist and accuse them of monopolistic and anti-competition practices by pushing out small players etc. At least that worked for me, the second time it happened to me.

I email a friend who was on hotmail, to double check my emails goes to spam and it did. I then used this as an example in my reply to Microsoft and did also what you suggested.

Lets see what they come back with.

FingerlessGlov3s commented 3 years ago

They've come back to me, they've going to look in to the issue along with the Escalations Team. So getting somewhere I hope.

patschi commented 3 years ago

issue along with the Escalations Team

Appears to be the new standard email template they're sending out. Let's see if that helps. Keep us updated!

FingerlessGlov3s commented 3 years ago

They've gotten back to me now.

your IP has been flagged by our system as suspicious, causing your IP to become blocked So they gonna remove the block and will take 24-48 hours to fully complete, then they said again to signup to JMRP and SNDS, they like to keep saying this even if you've told them twice you have :-| .

They then talked about Sender Score Certified Mail program (Sender Score) - https://www.validity.com/products/returnpath/certification/ Although not sure how easy to get this will be when your self hosting email for personal usage only. If it was for business usage small-medium it might be doable, I'm guessing there some form pricing evolved. Sender Score is the only service to which we subscribe. Microsoft Stated

Have emailed validity.com about the program and shall see where I get too.

They also linked this old White Paper from 2007? http://download.microsoft.com/download/e/3/3/e3397e7c-17a6-497d-9693-78f80be272fb/enhance_deliver.pdf

Side Note When I replied to Microsoft and got this reply about having my email signed, which I thought was funny. The following attachment(s) could not be processed by the Microsoft support system: OpenPGP_signature.

andryyy commented 3 years ago

Everything I've read are 100% the default reponses.

FingerlessGlov3s commented 3 years ago

@andryyy My thoughts exactly. Canned responses no doubt, wonder if they will unban my IP or if its all talk...

jol64 commented 3 years ago

they unbanned me only after I complained to privacy department, and it worked temporarily only. But I assume this is the only route that will work if we continue to keep them busy.. Unsure whether complaints to Irelandยดs data protection authority help - I am not Max... I also complained in Brussels about unfair practices but got a bla bla response. Maybe more complaints will help to get that going.

FingerlessGlov3s commented 3 years ago

I asked them how long till I stay "mitigated" and... unable to provide any details about this situation since we do not have the liberty to discuss the nature of the block Maybe because I'm not the ASN owner not sure... Guessing its just them playing the game.

Yet again they repeat the same thing they do in all emails. At this point, I would suggest that you review and comply with Outlook.com's technical standards. This information can be found at https://postmaster.live.com/pm/postmaster.aspx

EricThi commented 3 years ago

Hello, Do you have read my little "fix" on gafam ?

https://github.com/mailcow/mailcow-dockerized/issues/2851#issuecomment-686356475

at this time : no spam and no rejected mail on my side :)

All config are cleaned and unchanged

I check every day if my ip are on rbl (via rblmon.com, no check via api or another website)

FingerlessGlov3s commented 3 years ago

@EricThi, Have you tested Outlook accounts you've never sent emails to before? because I was thinking of doing those cron emails, but Outlook does per inbox HAM, it looks.

I get Index delivery on two Outlook accounts (Mine and Family Remember), so I got a email from a colleague of mine and boom went to SPAM, which proves per inbox HAM or contacts related HAM.

EricThi commented 3 years ago

@FingerlessGlov3s In first time, i have validate on my test account, when send mail directly on inbox, i have create new email and test it : Receive in Spam, search and found : miss send "many mail" all day => cron Last test : send to email already existing and check on live : Inbox directly

After, One of domains on my mailcow is used (by a teacher) for send mail to people (parents) with many providers emails : outlook/live/gmail/free(fr ISP) => no spam

After, i have change today some params on my dns zone (clean) and reduce spf : "v=spf1 mx a -all" Why ? because on this website : https://www.dmarcanalyzer.com/fr/spf-4/checker/ => to many include, spf can be skip by isp and include google and cie is useless...

After, i validate my spf via : https://www.kitterman.com/spf/validate.html?

send mail to check spf/dkim : http://www.open-spf.org/Tools/

and found best practice for send mail to google and office with spf and dkim : https://emailtrends.com/news/2020-dmarc-works-autumn-update/

I edit my previous post

FingerlessGlov3s commented 3 years ago

@EricThi, good stuff there, people recommend you use put the IPs directly in the SPF rather then using "a" or "mx". As some mail servers fail to do the mx and a lookups sometimes (less DNS lookups the better). Also MS don't use IPv6 nor TLS1.3, which I find little odd. Example SPF: "v=spf1 ip4:51.51.51.51 ip6:2001:41d0:800::1 -all"

My emails going to spam stopped after Microsoft removed that "block" (mitigation), clicking not spam personally didn't fix anything :-(, until Microsoft did this thing.

That email trends was getting different results too me, before I contacted MS, so I wonder if my "block" was my issue. Lets hope they don't "block" me again. Otherwise I'll have to email them again.

EricThi commented 3 years ago

@FingerlessGlov3s Set a or mx can be used for farm server mail for me. yes, for one server mailcow (or another) it's oversized :) IPV6 and tls1.3 it's not a standard at this time. On my side, i remove ipv6 on all hosts and my frontal FW block ipv6. I use tls1.3 by default with failback to tls1.2

On my side, i have request to microsoft to remove my ip too and unblock my domain, because I use a low cost ndd (.ovh personnal use, low renew) => I never thought that it's by "default" blacklisted...

At this time, if send an email to @hotmail.com or @outlook.com => Inbox.

After, I send "many" mail all day to Microsoft and Google for create traffic :)

FingerlessGlov3s commented 3 years ago

I use OVH services myself, but not looked in to getting their TLD yet though... Its super cheap though, compared to other TLDs. Should have better rep than a .tk tld atleast you'd think so. Which I have but unfortunately I used it free since 2011/2012 or something, so I can just moved off, but I just purchased it in 2014 and purchased it till 2028. It's one of the most popular tlds but not the most popular for spam some research suggests though. Now trying to move the registrar of the .tk domain is proving very hard, might take months to move it from freenom.

I wonder if MS block unseen IPs and domains (unless the IP is good reputation eg gsuite) by default to try help combat spam... Only thing I can think of but I still know people who get lots of spam enter their inbox on Outlook.

Have you also enabled DMARC reporting? So your mail servers send reporting emails for other domains?

apiraino commented 3 years ago
1. Bonus :

After mail are good (not send "spam" before contact delivery support xD) for help microsoft/google to get data, i have create 4 little crontab on my server to send mail via my mailcow to gmail and outlook address

and

After, I send "many" mail all day to Microsoft and Google for create traffic :)

@EricThi thank you for describing all the steps you took to whitelist your domain. I have a question about the above method to "feed" good emails to keep the reputation good. Do you have proof that it works? Any way to verify that?

To be clear, I have the same problem described in this issue (new mailcow installation, SPF1, DKIM, DMARC all pass). I'm not sending mail to a recipient on MS cloud service, rather Gmail. But the effect is the same: opaque heuristics to classify as spam, impossible to debug the issue, unhelpful answers from google.

EricThi commented 3 years ago

I use OVH services myself, but not looked in to getting their TLD yet though... Its super cheap though, compared to other TLDs. Should have better rep than a .tk tld atleast you'd think so. Which I have but unfortunately I used it free since 2011/2012 or something, so I can just moved off, but I just purchased it in 2014 and purchased it till 2028. It's one of the most popular tlds but not the most popular for spam some research suggests though. Now trying to move the registrar of the .tk domain is proving very hard, might take months to move it from freenom.

I wonder if MS block unseen IPs and domains (unless the IP is good reputation eg gsuite) by default to try help combat spam... Only thing I can think of but I still know people who get lots of spam enter their inbox on Outlook.

Have you also enabled DMARC reporting? So your mail servers send reporting emails for other domains?

Hello @FingerlessGlov3s for tld .ovh, it's a response by 4/5 websites to switch my old email to my new mail with .ovh => per default .ovh is blacklisted for create new account (or change email) because it's used by "many" spam. ==> blacklisted by website or by a federate security for webshop/anothers (example for me : my mobile operator...)

My ip is neutral on reputation (it's not poor or bad, just neutral because i send low volume) after, i don't have email reputation because no newsletter, just send direct email.

I have all report enable for dmarc to send on my main domain via : domaintoreport.tld._report._dmarc.maindomain.tld. TXT "v=DMARC1"

After, all report are 100% good, because my email are not reject/detected to spam, just mark as spam by a post rules on gafam...

Since yesterday, my email were tagged in spam to google.. I search the reason and for me it's linked by a bad RBL /:

=> Big impact on my "reputation" for google, tagged on spam :/ ==> same to outlook now... (for previous mail already send many email with scripts)

==> I have contact google and Microsoft for that. After, for microsoft on supervision : https://sendersupport.olc.protection.outlook.com/snds/data.aspx?wa=wsignin1.0 ==> all is green, no spam send <<

I have add 2 domain on postmaster google for test it : https://postmaster.google.com/ ==> waiting to send more 100 mails per day for get statistics ===> i use my old account, because i need to remove it for leave gafam :D

I waiting response by microsoft and google support now.

1. Bonus :

After mail are good (not send "spam" before contact delivery support xD) for help microsoft/google to get data, i have create 4 little crontab on my server to send mail via my mailcow to gmail and outlook address

and

After, I send "many" mail all day to Microsoft and Google for create traffic :)

@EricThi thank you for describing all the steps you took to whitelist your domain. I have a question about the above method to "feed" good emails to keep the reputation good. Do you have proof that it works? Any way to verify that?

To be clear, I have the same problem described in this issue (new mailcow installation, SPF1, DKIM, DMARC all pass). I'm not sending mail to a recipient on MS cloud service, rather Gmail. But the effect is the same: opaque heuristics to classify as spam, impossible to debug the issue, unhelpful answers from google.

@apiraino : yes, depend to robot/human on request support :/ https://toolbox.googleapps.com/apps/checkmx/ it's good for you .? At this time, my spf are "bad" for google <<

error SPF must allow Google servers to send mail on behalf of your domain. help_outline Help center article => https://support.google.com/a/answer/33786 Decision SPF fail - not authorized Record v=spf1 mx a ip4:my.ip.mail.server -all

and after follow many docs (very bad docs for me) : https://support.google.com/a/answer/33786#zippy=%2Crechercher-votre-fournisseur-de-domaine%2Cfournisseurs-de-messagerie-tiers%2Cenregistrement-txt-pour-spf%2Cfacultatif-v%C3%A9rifier-votre-enregistrement-txt-pour-spf-actuel

Good configuration need to return on spf :

_spf.google.com
_netblocks.google.com suivi de plusieurs adresses IP
_netblocks2.google.com suivi de plusieurs adresses IP
_netblocks3.google.com suivi de plusieurs adresses IP

After change my spf : "v=spf1 mx a ip4:my.ip.mail.server include:_spf.google.com include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com -all"

=> checkmx see only warning because :

warning No Google mail exchangers found. Relayhost configuration? help_outline Help center article => https://support.google.com/a/answer/33915 If you intentionally set up a mail server somewhere on your premises that automatically forwards all incoming mail to Google you may disregard this warning. Otherwise - this is a serious configuration error as it causes disruption of mail flow.

ok, it's for send directly via google ^^ ok, big joke by google...

After, i will not trust google on my spf... and for "good" practice, now, it's needed... When explain on my first comment, for google, if we have "good" rules (for google) it's ok... for MS, it's very different :/

Have fun

EricThi commented 3 years ago

I found this another check found on google support : https://www.checktls.com/TestReceiver (src : https://support.google.com/mail/thread/3973530?hl=en)

My result :

CheckTLS Confidence Factor for "mail@mydomain.tld": 100 all are ok at 100% [005.141] We can use this server [005.141] TLS is an option on this server [005.141] โ€‘โ€‘> STARTTLS [005.230] <โ€‘โ€‘ 220 2.0.0 Ready to start TLS [005.230] STARTTLS command works on this server [005.331] Connection converted to SSL SSLVersion in use: TLSv1_3 Cipher in use: TLS_AES_256_GCM_SHA384 Perfect Forward Secrecy: yes Certificate #1 of 3 (sent by MX): Cert VALIDATED: ok

I have too send request to google via answer no to all questions (and it's true) : https://support.google.com/mail/troubleshooter/2696779?visit_id=637468153468059253-3193426294&hl=en&rd=3#AppsHC

for get this url : https://support.google.com/mail/contact/bulk_send_new

apiraino commented 3 years ago

I found this another check found on google support : https://www.checktls.com/TestReceiver

Also tested that one, all is green for my mail server (as expected).

https://support.google.com/mail/contact/bulk_send_new

I have submitted a request to this form a couple of days ago. They should provide an answer in two weeks time (what a joke)

In my case, I don't use google in any form to send emails so all their tooling to check my "reputation" are mostly useless for me. I just need them to not greylist me for no apparent reason.

I'm sorry to read that because of OVH you've been blacklisted :-(

Anyway, thanks for your suggestions :+1:

Clete2 commented 3 years ago

@gromain did you try the routing option in the admin menu with sendgrid/mailgun/AWS SES & etc. of mailcow?

Mailcow

This works for me as a bandaid. I don't like using it, but I see no other choice.

I tried the fix that @EricThi laid out. I swapped to a new IP address, but unfortunately in the middle of implementing @EricThi 's scripts, I got banned on the NEW IP address from Microsoft. My emails were handwritten between my Mailcow installation and my personal MSN account.

Like others, I had no luck with Microsoft support. Not eligible for mitigation, and they won't discuss it with me.

Edit: I sent an absolutely scathing email to Hotmail support in response to their "sorry but we aren't doing anything" response accusing them of anticompetitive behavior and that I am telling all my clients to shutdown their Microsoft accounts. They responded by reopening my case, investigating, and unblocking me. Please let this whitelist stay.

Edit2: I implemented something similar to @EricThi, but used Enron's e-mails instead. I detailed my approach here.

mfld-pub commented 3 years ago
1. Bonus :

After mail are good (not send "spam" before contact delivery support xD) for help microsoft/google to get data, i have create 4 little crontab on my server to send mail via my mailcow to gmail and outlook address :

@hourly cp /usr/local/smtp/template-joke_jod /tmp/joke_jod && curl -X GET "https://api.jokes.one/jod" -H  "accept: application/json" -H  "content-type: application/json" -H  "X-JokesOne-Api-Secret: api_key" | jq '.. | .text?' | sed 's/null//g' | sed 's/\\r/ /g' | sed 's/\\n/ /g' >> /tmp/joke_jod && cat /tmp/joke_jod | msmtp -a joke your.email@outlook.com, your.email@gmail.com && sleep 2 && rm /tmp/joke_jod

@hourly cp /usr/local/smtp/template-joke_blonde /tmp/joke_blonde && curl -X GET "https://api.jokes.one/jod?category=blonde" -H  "accept: application/json" -H  "content-type: application/json" -H  "X-JokesOne-Api-Secret: api_key" | jq '.. | .text?' | sed 's/null//g' | sed 's/\\r/ /g' | sed 's/\\n/ /g' >> /tmp/joke_blonde && cat /tmp/joke_blonde | msmtp -a joke your.email@outlook.com, your.email@gmail.com && sleep 2 && rm /tmp/joke_blonde

@hourly cp /usr/local/smtp/template-joke_animal /tmp/joke_animal && curl -X GET "https://api.jokes.one/jod?category=animal" -H  "accept: application/json" -H  "content-type: application/json" -H  "X-JokesOne-Api-Secret: api_key" | jq '.. | .text?' | sed 's/null//g' | sed 's/\\r/ /g' | sed 's/\\n/ /g' >> /tmp/joke_animal && cat /tmp/joke_animal | msmtp -a joke your.email@outlook.com, your.email@gmail.com && sleep 2 && rm /tmp/joke_animal 

#Baconipsum
*/5 * * * * cp /usr/local/smtp/template-baconipsum /tmp/baconipsum && wget "https://baconipsum.com/api/?type=meat-and-filler&paras=5&format=text" -O /tmp/baconipsumtmp && cat /tmp/baconipsumtmp >> /tmp/baconipsum && cat /tmp/baconipsum | msmtp -a joke your.email@outlook.com, your.email@gmail.com && sleep 2 && rm /tmp/baconipsum*

This is an awesome idea to help boost sender reputation. I was curious why Google and MS postmaster tools never showed anything for my domains, even they are 2 years in prod. Turns out 250 mails a day is not enough. So about 2 weeks ago I implemented your suggestion but as of now postmaster tools still shows no data. Does this technique still work ?

I I send to 3 different gmail and 3 different microsoft accounts.

Google DMARC report for a typical day:

Ipv4

293

IPv6

307

Could it be that because I am dual stacked and each IP is treated separately the postmaster tools won't show any data for me because my ~600 a day are about 50/50 split IPv4 and IPv6 ?

Edit: I see in @Clete2 writeup from the comment above

I have 288 randomly selected e-mails being sent to Microsoft servers each day that are actively read by a user.

In my case I just have a filter rule to move them to a folder and mark as read. Are they tracking that and disregarding the messages ? I log into my receiving accounts every now and then to reply to bacon ipsum mails to show "interaction" of sorts.

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.