search

virtualmin / virtualmin-gpl

Virtualmin web hosting control panel for Webmin
https://www.virtualmin.com
GNU General Public License v3.0
304 stars 95 forks source link

TLS SNI matching domain certs for Postfix SMTP, Dovecot IMAP and ProFTPd #21

Closed chris001 closed 4 years ago

chris001 commented 6 years ago
jcameron commented 6 years ago

This should be already happening in the latest release for Dovecot, for new domains or domains when SSL is enabled. However, I don;t know of any way to configure Postfix or ProFTPd to work this way - they only support per-IP certs.

chris001 commented 6 years ago

cPanel offers this feature (unique SSL cert per domain for Web, SMTP, IMAP, and FTP), so Virtualmin could and should also. Here's a page with the command line tests used to verify whether or not a Virtualmin virtual server's IMAP or SMTP service is using the matching cert for the virtual server domain name. https://www.servermule.com.au/help/cpanel/testing-cpanel-sni-ssl-mail-exim-dovecot/

1 Testing Dovecot SNI TLS/SSL Request

How to test Dovecot is using the correct certificate with an SNI request:

openssl s_client -connect localhost:993 -servername mail.myvirtualserver.com

The “SNI Name” is the name of your customer’s domain mail server, so for example, mail.yourcustomer.com

2 Testing Exim SNI TLS/SSL Certificate

openssl s_client -tls1 -starttls smtp -connect localhost:587 -servername mail.myvirtualserver.com

As above, the “SNI Name” is the name of your customer’s domain mail server, so for example, mail.yourcustomer.com

More about AutoSSL, SNI and cPanel’s SSL developments

OMG, cPanel now supports using SNI SSL certs for mail services that are generated using AutoSSL!

Wait, what? OK here’s why you should be excited.

For a long time if you wanted to use TLS/SSL encryption on domains hosted on cPanel you had two options. 1) Secure the name of your server with an SSL certificate and then get all your clients to use yourserver.yourdomain.org as their server name OR 2) use the self signed cert and make them accept an error on each connection.

Not ideal.

But now cPanel supports SNI SSL certificates for exim and dovecot, which means you can use a different certificate for each of your domains without having to have a dedicated IP for each site.

I know what you’re thinking, that’s awesome, but I still need to buy an SSL certificate for each of them… This is where AutoSSL comes into play. cPanel now issues a cPanel (backed by comodo) certificate for EACH domain on your server.

chris001 commented 6 years ago

Another page with details on how to test Virtualmin is using the matching SSL certs for each virtual server domain name you try to connect to: https://support.context.io/hc/en-us/articles/115001711946-Troubleshooting-SSL-certificates-for-IMAP-providers

  1. Connect with SSL to all the SMTP/IMAP server ports.

openssl s_client -connect mail.myvirtualserver.com:143 openssl s_client -connect mail.myvirtualserver.com:465 openssl s_client -connect mail.myvirtualserver.com:587 openssl s_client -connect mail.myvirtualserver.com:993

  1. Inspect the certificate. Things to look for: a. Under “END CERTIFICATE” you should be able to see who issued it, and what names are on the certificate. b. “No client certificate CA names sent” means a CA name should be added to the certificate (and it should match the IMAP server you are trying to connect to) c. Under “issuer” you should see a known third party issuer d. If under “subject” you see a different “Wildcard/CN” than the IMAP server you’re trying to connect to, this is likely a mismatch
jcameron commented 6 years ago

@chris001 - I understand how to test this, but as far as I know there is no way to configure Postfix to support multiple SSL certs on the same IP address (unlike Apache and Dovecot).

chris001 commented 6 years ago

OK. I guess cpanel is using Exim not Postfix, if Exim supports SNI, this would explain why they can provide the exact matching expected TLS cert for the domain.

It's just a whole lot less "spammy" to have LE domain validated matching certificates on all mail, web, and ftp services.

chris001 commented 6 years ago

About enabling TLS SNI for Dovecot, as it should be...

  1. Why does Virtualmin NOT write the Dovecot SSL settings to /etc/dovecot/conf.d/10-ssl.conf ? Instead it throws the SSL settings into the main conf file at /etc/dovecot/dovecot.conf.

    The most important SSL settings are (in conf.d/10-ssl.conf): From: https://wiki.dovecot.org/SSL/DovecotConfiguration

  2. How to configure Dovecot for SNI TLS. It's very easy.

    local_name imap.example.org {
ssl_cert = </etc/ssl/certs/imap.example.org.crt
ssl_key = </etc/ssl/private/imap.example.org.key
}
local_name imap.example2.org {
ssl_cert = </etc/ssl/certs/imap.example2.org.crt
ssl_key = </etc/ssl/private/imap.example2.org.key
}
# ..etc..

EDIT:

  1. Why is /etc/dovecot/dovecot.conf missing the line: !include conf.d/*.conf
  2. If this line (above) were to added to dovecot.conf, would Virtualmin fail to recognize this fact, and process the conf files wrong ?
chris001 commented 6 years ago

Anyone? Well..... To answer my own question...

  1. If you include ALL files in conf.d by adding !include conf.d/* to the end of dovecot.conf, then dovecot suddenly starts seeing the inbox metadata differently, causing there to appear to be no Inboxes/Trash/Spam/Junk Mail/etc when seen from inside the imap client mail app. This is because there are some uncommented, active lines in some of the default conf files located in conf.d, and these lines contain setting which conflict with and override some of the settings in Virtualmin's dovecot.conf. So you have to only !include conf.d/10-ssl.conf if you want to use this file and edit it and keep all the SSL and TLS SNI related settings in it organized in one place.
chris001 commented 6 years ago

To get TLS SNI to work in Dovecot IMAP mail clients. Add this functionality to Virtualmin:

  1. Scan the SSL TLS (ideally Lets Encrypt, not self-signed) certificates used for each mail domain, and extract the fully qualified domain names from each certificate.
  2. For each cert, for each domain name, add these four lines to /etc/dovecot/conf.d/10-ssl.conf: 
    local_name my.fullyqualifieddomainname.com {
ssl_cert = </home/server3/domains/my.fullyqualifieddomainname.com/ssl.cert
ssl_key = </home/server3/domains/my.fullyqualifieddomainname.com/ssl.key
}

    The file can get pretty big.

    • It would probably be best to generate all these lines of config text into its own auto-generated file which will be touched by nothing else other than Virtualmin, name it something like 10-sni.conf and save it in conf.d.
  3. Make sure these lines are added to dovecot.conf: 
    !include conf.d/10-ssl.conf
!include conf.d/10-sni.conf
chris001 commented 6 years ago

Big question is:

  1. Where does Virtualmin expect you to put certificates and keys? It seems to have some undocumented, yet rigid, rules about how and where certificates can be located, or else, fail errors.
  2. Why does a symlink to the cert, in the home folder for a virtual server, cause a fail, when you click on "Copy cert to Postfix", "Copy cert to Dovecot", "Copy cert to ProFTPD" ?
  3. EDIT: Isn't there any distribution-independent standard locations for storing certs keys and CA-certificate chains? Where does cpanel, directadmin, etc, all securely store domain certs keys and CA-cert chains? Storing these in the home folder of the virtual server, with ownership set to the domain admin, and permissions set to 0700, seems to be a huge security hole, isn't it supposed to be root:root and 0400?
jcameron commented 6 years ago

  1. Virtualmin expects that certs and keys go in each domain's ~/etc directory, and be regular files rather than symlinks.

  2. A symlink may cause problems if the domain owner cannot read the file it it's new location. I'd recommend just sticking with Virtualmin's default locations to avoid unpredictable problems like this.

  3. I don't know of any standard location for certs and keys. Long-term we'd like to move them out of home directories though, as there is a risk that a user could prevent Apache from starting by corrupting their cert.

swelljoe commented 6 years ago

/etc/pki is the standard location for keys on RHEL/CentOS. We could create a virtualmin directory (there's already a few directories there for other services...rpm, java, rsyslog, etc). Though there's also a tls directory, which I guess might also work, since we're talking about sharing them across a bunch of services, so it's not really specific to Virtualmin.

I think Debian/Ubuntu uses /etc/ssl somewhat similarly.

Here's how the go folks handle finding certs: https://golang.org/src/crypto/x509/root_linux.go

jcameron commented 6 years ago

Whatever we do, it's probably best to have a standard location across all Virtualmin installs, which means somewhere under /etc/webmin

chris001 commented 4 years ago

I was going to make a new issue. This issue is still relevant today. Follows up to issues webmin/webmin#913 and webmin/webmin#503 Also following up on forum post: https://www.virtualmin.com/node/65711

jcameron commented 4 years ago

Thanks, I didn't know that Postfix supported SNI now! Do the majority of mail clients also support it though?

chris001 commented 4 years ago

Thanks, I didn't know that Postfix supported SNI now! Do the majority of mail clients also support it though?

Yes, SNI is supported by most if not all the top market share email client software:

Note, It's easy for mail client software to support SNI, they just pass the hostname of the mail server they're attempting to connect to, as an additional parameter to OpenSSL, which has supported SNI TLS connections since 2007, version 0.9.8: https://en.wikipedia.org/wiki/Server_Name_Indication

jcameron commented 4 years ago

Also, does it matter if other mail servers that might be sending to your domain support it? Could you run into a situation where email bounces because the sender's SMTP server doesn't see the correct cert?

chris001 commented 4 years ago

Also, does it matter if other mail servers that might be sending to your domain support it? Could you run into a situation where email bounces because the sender's SMTP server doesn't see the correct cert?

Yes, it's possible, when you have your mail server configured to do only encrypted MTA-to-MTA conversations, when/if Virtualmin serves an expired cert or non-matching hostname on the cert, should cause the remote MTA to quit and drop the connection, so whichever incoming email message would be undelivered, then the remote MTA will retry the email message delivery later, according to the default retry algorithm schedule.

To give the server or domain admin notification of these undelivered email messages, there's two standard specs/ RFCs one uniformly handles encrypted mail delivery by publishing which hostnames and ports the domain accepts TLS encrypted sessions for incoming email, and the other standard reports on TLS delivery failure: Every day, each supporting mail provider (gmail for example) sends an email, to your domain's specified email address, a "TLS Report", on how many emails it delivered to you, and how many it failed to deliver, with each reason why, eg expired cert, non-matching name on cert., compatible cipher not found, etc These two standards are called MTA-STS and TLSRPT.

Here's how easy it is to automate setup MTA-STS and TLSRPT on each domain:

  1. https://support.google.com/a/answer/9276511?hl=en
  2. https://www.digitalocean.com/community/tutorials/how-to-configure-mta-sts-and-tls-reporting-for-your-domain-using-apache-on-ubuntu-18-04
jcameron commented 4 years ago

Ok, we'll add this to our TODO list.

chris001 commented 4 years ago

This is a great checker that inspects the domain and tells you, of all mail-related security features, which are configured right/wrong, and why. It should be great help in validating whether the code is working: https://en.internet.nl/test-mail/

chris001 commented 4 years ago

And this great checker gives you a score from 1 to 10, based on spammyness of your outgoing email ie our Virtualmin system's Postfix security config. https://www.mail-tester.com/ From testing, Virtualmin here scored 4.0/10 because DNSSEC, DANE, DKIM, and connect over IPv6, are currently broken. Virtualmin Postfix config should/must score 10/10 and achieve the best rating possible for outgoing and incoming mail security.

chris001 commented 4 years ago

While testing spammyness of email, the result is: DKIM is totally failing to sign outgoing email messages thru Postfix! Webmin/Virtualmin's OpenDKIM install/config isn't actually running DKIM? When you send an email thru the Virtualmin server, you get this error in /var/log/mail.err: Oct 13 17:43:32 server1 dkim-filter[896]: 4163A1A045D: no signature data

Here's the step by step install config of OpenDKIM, to verify the config steps: https://www.linode.com/docs/email/postfix/configure-spf-and-dkim-in-postfix-on-debian-8/

jcameron commented 4 years ago

That looks like an openDKIM configuration issue if signing isn't happening. Unfortunately every distro and version sets up their packages differently ... which are you running there?

chris001 commented 4 years ago

That looks like an openDKIM configuration issue if signing isn't happening. Unfortunately every distro and version sets up their packages differently ... which are you running there?

Debian 8 Jessie. Q: Shouldn't Virtualmin complain/surface an error message to the sys admin on the main web interface page, when any enabled and expected working security subsystem is in a fail state? Such as:

  1. when DKIM outgoing email signature is failing/does not exist,
  2. DNSSEC zone is invalid,
  3. DANE is invalid (depends on DNSSEC valid) ?
chris001 commented 4 years ago

Further debugging work. Part of the problem is that the custom DKIM keys, in bind9 DNS for each of the domains, used by foreign SMTP servers to verify each email's DKIM signature, doesn't match the dkim.key default key, configured by Virtualmin for opendkim to use to sign each domain's email messages. To be clear: Virtualmin configured opendkim to always use the default DKIM key, even on domains set to use a custom DKIM key. Run the command: opendkim-testkey -d mydomain.com -s _**DKIMkeyTagYear**_ -vvv For example, on this server: opendkim-testkey -d mydomain.com -s 2014 -vvv The output is fail:

opendkim-testkey: using default configfile /etc/opendkim.conf
opendkim-testkey: key loaded from /etc/dkim.key (PROBLEM conf says use default key dkim.key)
opendkim-testkey: checking key '2014._domainkey.mydomain.com' (Read custom key from DNS)
opendkim-testkey: key not secure (DNSSEC = fail)
opendkim-testkey: keys do not match (default key !== custom key)
iliajie commented 4 years ago

Jamie, do we have full SNI support for Dovecot and Postfix already?

chris001 commented 4 years ago

Jamie, do we have full SNI support for Dovecot and Postfix already?

I see some SNI for Dovecot: the local_name config setting in Dovecot config: Code: https://github.com/virtualmin/virtualmin-gpl/search?q=local_name&type=Code Issue: https://github.com/virtualmin/virtualmin-gpl/issues/91

chris001 commented 4 years ago

The Postfix SNI code is missing.

1.

----- /etc/postfix/main.cf -----


# provide the primary certificate for the server, to be used for outgoing connections
smtpd_tls_chain_files =
/etc/letsencrypt/live/servername.serverdom.com/privkey.pem,
/etc/letsencrypt/live/servername.serverdom.com/fullchain.pem

provide the map to be used when SNI support is enabled

tls_server_sni_maps = hash:/etc/postfix/vmail_ssl.map

-----

2.
----- `/etc/postfix/vmail_ssl.map` -----

Compile with postmap -F hash:/etc/postfix/vmail_ssl.map when updating

One host per line

servername.serverdom.com /etc/letsencrypt/live/servername.serverdom.com/privkey.pem /etc/letsencrypt/live/servername.serverdom.com/fullchain.pem servername.otherdom.com /etc/letsencrypt/live/servername.otherdom.com/privkey.pem /etc/letsencrypt/live/servername.otherdom.com/fullchain.pem


-----

3. Then run the command:
`postmap -F hash:/etc/postfix/vmail_ssl.map`

4. Restart postfix as normal:
`service postfix restart`

5. Test. Run:
`openssl s_client -connect localhost:25 -servername servername.otherdom.com -starttls smtp`
`openssl s_client -connect localhost:25 -servername servername.serverdom.com -starttls smtp`
You'll find the hostname under the certificate details. 
It will match the default server name of the host if there is NOT a match. 
Be sure the server name of the host is in the map file (`vmail_ssl.map`) for that reason.
iliajie commented 4 years ago

Ref.: https://forum.virtualmin.com/t/virtualmin-plans-for-new-postfix-version-3-4x-and-sni/104211/5

jcameron commented 4 years ago

So I'm working on implementing this the way @chris001 suggested - but which Linux distributions are supplying a Postfix 3.4 package already? The docs suggest that the smtpd_tls_chain_files directive is only supported in that version..

iliajie commented 4 years ago

but which Linux distributions are supplying a Postfix 3.4 package already

Postfix 3.4.x is provided already by Debian 10, Ubuntu 20.04 and Fedora 32. Support for latter two is being added in the near future - working on it right now.

chris001 commented 4 years ago

Rather than check the OS Distro name and version number, it'd be far better to run the following command: postconf mail_version to check if the version of postfix is greater than or equal to 3.4, because any system admin can build postfix 3.4 from source and install it, to replace the distro supplied version, to get the new SNI feature in 3.4+. When 3.4+ is detected, then, add the SNI config lines to the postfix config file.

On a debian 8 system, with default distro postfix, the command postconf mail_version returns multiple lines:

mail_version = 2.11.3
postconf: warning: /etc/postfix/main.cf: unused parameter: smtpd_host_lookup=dns
postconf: warning: /etc/postfix/main.cf: unused parameter: smtpd_dns_support_level=dnssec
jcameron commented 4 years ago

Yes, the code will check the actual Postfix version. I was wondering how many distros come with Postfix 3.4 by default, for my own testing.

iliajie commented 4 years ago

I think at least Debian 10 and Ubuntu 20.04. Fedora Server 32 would be also great!

chris001 commented 4 years ago

Yes, the code will check the actual Postfix version. I was wondering how many distros come with Postfix 3.4 by default, for my own testing.

  1. Debian STABLE 10 (Buster) comes with Postfix 3.4.
  2. Fedora 30+
  3. OpenSuse Leap 15.2
  4. OpenSuse Tumbleweed
  5. Arch
  6. FreeBSD 11 and 12.
  7. Centos 7 and 8 ONLY IF you add the 3rd party repo "ghettoforge" and yum install the postfix3 related packages from there.
jcameron commented 4 years ago

Ok, this has FINALLY been implemented, and will be included in the next Webmin and Virtualmin releases.

iliajie commented 4 years ago

A forum post for those who is willing to try to install Postfix 3.4+ with SNI support on CentOS 7/8.