virtualmin / virtualmin-gpl

Virtualmin web hosting control panel for Webmin
https://www.virtualmin.com
GNU General Public License v3.0
312 stars 97 forks source link

To improve virtualmin outgoing email deliverability, run proactive checks and alert admin. #561

Open chris001 opened 1 year ago

chris001 commented 1 year ago

It's simple and dependable to utilize external outgoing email forwarders such as gmail or mailgun.

However, for a virtualmin server to give all of its emails to an outside party ("Man In The Middle"), those external companies will mine data from the contents of the emails, and do anything with your data to make a profit. That sacrifices privacy, which is unacceptable for most virtualmin users and organizations, and should be for the virtualmin org.

virtualmin-gpl software could, and should, be made equally as strong on outgoing email as google gmail, mailgun, etc, in its outgoing email, try harder to be sure it will deliver messages reliably into the receiver's inbox, resilient against blacklisting on RBL denylists.

External forwarders such as mailgun should be a distant third place fallback sending method, in case of emergency, when virtualmin detects a bounce, and the email message is urgent priority, and when the email messages contain no personal or private information e.g. auth codes, password reset URLs, etc.

There should be a feature "Diagnostic Check - Outbound Email" to periodically auto check the outbound email server for best practice configuration which maximizes the resilience against IP blacklist:

  1. Check PTR records correctly map to the hostname of each domain's outgoing mail server's IP addresses.
  2. Check DKIM is enabled and working to sign all outbound emails with DKIM, for all domains.
  3. Check SPF records are both enabled and correctly specified in the DNS records, for all domains.
  4. Check DMARC enabled and working for all domains.
  5. Check DANE, if any, is enabled correctly in postfix conf and in DNS records, for all domains.
  6. Check TLSA records are serving working keys for all mail related ports, with all domains SNI.
  7. For outgoing email, default footer Ask the recipient to add your domain name to their approved senders list (whitelist). And if the recipient’s mail service doesn't support approved senders, the recipient's IT admin should add the domain's sending IP addresses to an allowlist. Extract the current range of sending IP addresses from the domain's SPF record.
  8. Every hour, or other interval, poll the well known public RBL denylists for each of the virtualmin server's domain's outgoing SMTP IPv4/IPv6 addresses, then send automated email message, every day, to those RBL denylist service admins, ask to remove the virtualmin server's outgoing IP addresses from that RBL denylist, until the outgoing IP addresses are removed from RBL denylists.
  9. Check outgoing email message for spam and antivirus and when detected, drop message.
  10. Any other check needed to keep up the reputation of all the domains' outgoing mail MX records and IP addresses.
jcameron commented 1 year ago

That's not a bad idea - email deliverability is a real issue for people running small sites. Sadly I've seen cases where ever when all those requirements are met, some recipients still reject email for unknown reasons (ie. iCloud).

Other than RBLs, is there a better service to check for deliverability?

chris001 commented 1 year ago
  1. Sender Score is a free tool from Return Path – maker of other tools for tracking and improving deliverability rates. Sender Score rates IP addresses on a scale of 1 to 100, with 100 indicating a perfect score.

It doesn’t take into consideration as many factors as other tools do, but it looks into the most important ones.

You type your email domain or IP address on the homepage. Then you’ll be asked to provide basic information, your full name, work email, monthly volume, and country.

Once you hit the ‘What’s my score’ button, you’ll see information about DNS records, SSL certs, and the list of sending IPs with their respective sender scores.

A score of 80 or more is considered good. If you’re below 80, there’s room for improvement. You’ll notice a drop in your delivery rates when your score reduces even slightly.

If you register, you’ll see the ratings from the IP addresses you use for sending, as well as for related domains (domains sending email from, or web hosted on, the same IP as yours?). This score is one of the key things spam filters take into consideration when assessing whether an email should be delivered into the destination user's Inbox, or redirected into their Junk/Spam box.

Sender Score doesn’t give you any tips on how to improve the score and it’s also far from trivial. It sometimes takes years to build up a good Sender Score from scratch. Of course, when sending via an external email sender service, you use their domain rating, not yours, at the huge sacrifice of your users' privacy and making your organization's security vulnerable to spies and unknown malicious parties. Which is unacceptable for most informed and conscientious individual users and organizations.

  1. Talos Intelligence from Cisco is a free tool for assessing your domain and IP reputation score in real-time. For simplicity, your reputation will be represented with one of three scores: Good, Neutral, or Poor.

‘Good’ indicates that the harmful behavior associated with this address is negligible at most. ‘Neutral’ means that you are still on a reasonable level but some messages may be filtered out. ‘Poor’ represents poor deliverability and probable issues with inbox placement.

Talos also gives insights into the spam level associated with your IP address, its presence on blacklists, and its email volume.

Note: If you’re mainly using subdomains to send emails, typing in the domain name may return limited results. Try searching for each subdomain to get accurate info.

  1. Microsoft Smart Network Data Services (SNDS) gives you the data to understand and improve your sender reputation for delivering your mails to users at Outlook.com . Hundreds of millions of Microsoft 365 (formerly Office 365) users are protected by this very tool.

SNDS is does more than monitor domain email sender reputation. It helps IP address owners (web and mail server admins) to detect compromised servers, malware, viruses, and botnets. SNDS helps network administrators detect these problems so that they can clean them up and make the internet a safer place.

  1. Google Postmaster Tools has an enormous amount of data about email deliverability. It’s a great tool to understand how Google will treat your email sending domain when your email message arrives at Google's incoming mail servers.

To use Postmaster Tools, you’ll need to have a high sending volume. You’ll also need to add custom records to your DNS to get started. Then you’ll be able to see how your IP and domain reputations fluctuate on a 4-level scale (bad, low, medium/fair, high).

Google will also let you know how you’re doing in terms of authentication, spam reports, delivery errors, and encryption.

  1. Send Forensics is a paid tool, with a 14-day free trial. It focuses on deliverability by allowing you to analyze content, link quality, sender email authentication protocols, sending domain, and IP.

When you register, you’ll be assigned an individual email address that you can send emails to. It’s not tied to the address you signed up with. Your organization can start sending emails to it right away.

When you craft an email and send it to that assigned unique analysis email address, the site will then assess different aspects of your email message, and give you a score for each aspect.

SendForensics compares your results with those of thousands of other companies using their service. It then tells you where you rank and in which fields you should improve.

There’s additional data in the paid plans. A detailed analysis of content, with suggestions of words to use. Deeper insights into reputation, inbox placement and an option to preview emails for popular email apps.

Send Forensics can be connected with Google Postmaster and Microsoft SNDS tools.

  1. Mail-tester.com is a freemium tool for analyzing specific emails and their content. To use it, you send an email to the given email address you see on the screen and hit the button.

In a few seconds after you send an email, you have a score ready. Mail-tester looks into various factors that affect deliverability. It checks the likelihood of it going to spam and gives tips on what to improve. It analyzes your domain for authentication methods (SPF, DKIM, DMARC) and reads the HTML code of your message, looking for things to improve. Finally, it checks if your domain made it to some blacklist recently which might affect your odds.

  1. Spamcheck offers a free tool for testing email deliverability.

Spamcheck you paste your email’s full headers and full contents into the dialog box. You’ll get back a score in seconds; the lower, the better. And you will get comments about how to improve the email, addressing:

The quality of links in the content
Authentications of the sender domain
Text to image ratio (certain proportions trigger spam filters)
The general quality of HTML 

Like Mail-Tester, Spam Check uses the popular SpamAssassin to provide its assessment, so it’s reliable.

Postmark also allows you to integrate Spamcheck into your app and automatically run spam checks on all outgoing emails. For this purpose, they offer their API for free (even if you don’t use Postmark service to send your emails). Libraries for integrating with Perl, Python, and others are available.

  1. GlockApps. To try this tool, you send your email to an address displayed on the homepage and get back the results in a few seconds.

GlockApps emulates email-sending through Postfix and two other platforms for sending emails. It then estimates where your email would land. You'll see, spam is not the only potential problem. A seemingly well-written and formatted email would skip most of the inboxes if sent with Mailgun! Lots of Gmail users would probably miss it too, because it would get buried under "other promotions".

The power of GlockApps is buried in the full reports available after registering (only the first three are free). Follow the instructions on the screen to send your test email to a carefully crafted list of recipients.

In a few seconds you should see your report.

You see lots of stats, same as from other tools. Since you sent an email to dozens of different domains spread between countries, you can also see how each performed. It’s useful when you know which accounts the majority of your contacts use. You can then predict how their mail app will react when exactly this message arrives, this time to a real inbox.

  1. MX Toolbox offers tools aimed to help improve the domains’ email sending capabilities. One of them is aimed at validating a domain’s authentication methods.

All you need to do is send your test email message to ping@tools.mxtoolbox.com - in a few minutes, you’ll receive an email back, with a link to your deliverability report. In this, you'll see if your domain has the essential authentications, and if they’re configured properly.

  1. Other options: Many Email Sending Providers (ESP) promote various features for testing deliverability: Mailgun, and Postmark (above). However, these tools are built into their respective platforms, so they're not recommended as stand-alone tools.

The Basics, without which, reliable deliverability is difficult

Proper authentication protocols will prove to all the incoming SMTP servers on the internet, that you, the sender, are who you say you are, the genuine sender, authorized to send on that domain and IP address, and with that TLS cert. In other words – you make it very hard for spammers or phishers to get away with impersonating you, and send malicious emails, as if they were from you, resulting in huge damage to your virtualmin mail server's sender reputation.

The standard set of authentications is (1) SPF, (2) DKIM, and
(3) the powerful DMARC.

Plus, it's essential for your SMTP MTA's (Postfix) MX record also have:

(4) PTR record(s) which resolve to your mailserver's hostname as it shows in its EHLO/HELO greeting banner, to other mail servers. Spam filters check if the SMTP banner greeting hostname matches your mail server IP's PTR record hostname, and deduct many points when mismatched! (5) The hostname from your email server IP's PTR record must match to one of the hostnames in the TLS cert your SMTP server uses during SNI negotiation and STARTTLS or SSL encryption.
(6) Enabling TLS/SSL encryption on all SMTP traffic helps, and (7) DNSSEC, TLSA and DANE, powerfully protect your sender score. These prevent scammers from performing a cache poisoning attack against a vulnerable remote caching DNS server administered by an ISP, which the spammer then uses to impersonate your highly trusted email sending domain, to send victims fully authenticated email messages containing spam malware and phishing attack emails, to all customers of that ISP, causing data losses, data theft, downtime, reports to RBLs, and ruin your clean sender score that took you many many hours of work to build up.