relaying: mails are not relayed with implicit tls (465) / improve scripts & configuration

Miscellaneous first checks

[X] I checked that all ports are open and not blocked by my ISP / hosting provider.
[X] I know that SSL errors are likely the result of a wrong setup on the user side and not caused by DMS itself. I'm confident my setup is correct.

Affected Component(s)

Mails are not relayed with implicit tls (465)

What happened and when does this occur?

DEFAULT_RELAY_HOST=[relay.smtp.isp.tld]:465
RELAY_HOST=relay.smtp.isp.tld
RELAY_PORT=465
RELAY_USER=MYUSERNAME
RELAY_PASSWORD=MYCOMPLICATEDPASSWORD

What did you expect to happen?

Mail should be forwarded through the relay.

How do we replicate the issue?

1. mailserver.env:

DEFAULT_RELAY_HOST=[relay.smtp.isp.tld]:465
RELAY_HOST=relay.smtp.isp.tld
RELAY_PORT=465
RELAY_USER=MYUSERNAME
RELAY_PASSWORD=MYCOMPLICATEDPASSWORD

Will complain about smtp_tls_wrappermode = yes and smtp_tls_security_level = encrypt. Added the following to docker-data/dms/config/postfix-main.cf
```
smtp_tls_wrappermode = yes
smtp_tls_security_level = encrypt
```
Sendmail from the cli works. sendmail -F "Firstname surname" -f "firsname@domain.tld" -t testmail@gmail.com Subject: Hello World

Sending from a smtp client does not.

warning: smtp_tls_wrappermode requires "smtp_tls_security_level = encrypt" (or stronger)
to=<....>, relay=none, delay=0.33, delays=0.33/0/0/0, dsn=4.3.0, status=deferred (server unavailable or unable to receive mail)

DMS version

v11.0.0

What operating system is DMS running on?

Linux

What instruction set architecture is DMS running on?

x86_64 / AMD64

What container orchestration tool are you using?

Docker Compose

docker-compose.yml

services:
  mailserver:
    image: docker.io/mailserver/docker-mailserver:latest
    container_name: mailserver
    # If the FQDN for your mail-server is only two labels (eg: example.com),
    # you can assign this entirely to `hostname` and remove `domainname`.
    hostname: mail
    domainname: domain.tld
    env_file: mailserver.env
    # More information about the mail-server ports:
    # https://docker-mailserver.github.io/docker-mailserver/edge/config/security/understanding-the-ports/
    # To avoid conflicts with yaml base-60 float, DO NOT remove the quotation marks.
    ports:
      - "25:25"     # SMTP  (explicit TLS => STARTTLS)
      - "143:143"   # IMAP4 (explicit TLS => STARTTLS)
      - "465:465"   # ESMTP (implicit TLS)
      - "587:587"   # ESMTP (explicit TLS => STARTTLS)
      - "993:993"   # IMAP4 (implicit TLS)
    volumes:
      - ./docker-data/dms/mail-data/:/var/mail/
      - ./docker-data/dms/mail-state/:/var/mail-state/
      - ./docker-data/dms/mail-logs/:/var/log/mail/
      - ./docker-data/dms/config/:/tmp/docker-mailserver/
      - /etc/localtime:/etc/localtime:ro
      - /etc/letsencrypt:/etc/letsencrypt:ro
    restart: always
    stop_grace_period: 1m
    cap_add:
      - NET_ADMIN
      - SYS_PTRACE

Relevant log output

No response

Other relevant information

To get it to work I had to add `-o smtp_tls_wrappermode=no` to `smtp-amavis` and `127.0.0.1:10025` in master.cf.

What level of experience do you have with Docker and mail servers?

[ ] I am inexperienced with docker
[ ] I am inexperienced with mail servers
[ ] I am uncomfortable with the CLI

Code of conduct

[X] I have read this project's Code of Conduct and I agree
[X] I have read the README and the documentation and I searched the issue tracker but could not find a solution

Improvements to this form?

No response

So the problem boils down to: the option smtp_tls_security_level is set in multiple locations and by different scripts. Strictly speaking, this is just non-optimal scripting / configuration from before my time here, but definitely fixable.

Searching for smtp_tls_security_level, I can see that is is dealt with here:

https://github.com/docker-mailserver/docker-mailserver/blob/0d30b92a83118e7c1cea7cdc40007a3f41d4fe6c/target/scripts/helpers/relay.sh#L84-L95

@polarathene wrote the new and much improved wrappers I believe. Can you please have a look @polarathene and see whether we can make the relay host setup more uniform? We may be able to use sed for Postfix's master.cf?

A question from my side to @docker-mailserver/maintainers: We currently use smtp_tls_security_level = may as the default in main.cf and elsewhere. Why don't we use enforce here? I assume we require STARTTLS (I mean encryption with STARTTLS) anyway, even when starting on port 25, right? Were does this happen, and why don't we use smtp_tls_security_level = enforce as a default?

This appears to be a configuration issue. I've managed to setup 3 docker-mailserver instances that correctly handle:

A - Origin (MTA forwarding mail to send to the relay service)
B - Relay (MTA that acts as a relay for A, eg SendGrid)
C - Destination (MTA that receives the relayed mail from B)

What I did notice was that some configuration with Postfix here was not always obvious when experimenting with changes. Eventually it would pick them up, but for more consistent testing supervisorctl restart postfix after applying config changes would better ensure your tests were not stuck on older configuration.

I did not need to mess with amavis services in master.cf, but I did choose to add a new service there as a relay transport. It's not required if you only intend to deliver outbound to 465, then keeping with smtp_tls_wrappermode = yes + smtp_tls_security_level = encrypt in main.cf is fine.

This was actually my first time setting such up, so I'll try provide a reproduction shortly to share here. I just wanted to provide a heads up due to activity taking place.

Once I've taken care of that I'll go over discussion taking place and respond to that :)

Apologies for taking so long to follow-up. I ran into some networking issues when making a revised reproduction config :sweat_smile:

I've tried to cover as much as possible, especially with the reproduction config. Feel free to ask for more information if needed :)

TL;DR

As @edge90 points out, issue is with Amavis (AFAIK, only affects smtp-amavis (smtp) not 127.0.0.1:10025 (smtpd)):

smtp-amavis just needs to explicitly opt-out of smtp_tls_wrappermode like it does with smtp_tls_security_level. Easy fix.
main.cf can then enable implicit TLS (465) for all outbound SMTP connections. Presently the relay setup won't detect implicit TLS from relay ENV, so the user must know to provide this additional config (does not appear to be documented?).

See the following message below for more details and reproduction.

Fixing outbound SMTP support for using implicit TLS

master.cf configures Amavis with the following services: smtp-amavis/unix (smtp) and 127.0.0.1:10025 (smtpd).
Setting main.cf:smtp_tls_wrappermode=yes fails as smtp-amavis service inherits this while explicitly overriding a parameter with smtp_tls_security_level=none instead of encrypt (already set by us in main.cf when a relay is configured).
The service does not seem compatible with upgrading that to encrypt however. Nor should it need to be as it's only meant to send locally within the container. Rather than inheriting default smtp_tls_wrappermode=no, it should also explicitly set that to avoid the issue.

Current workaround can run the following via running container or leverage user-patches.sh, alternatively these overrides can be provided with config/postfix-main.cf and config/postfix-master.cf:

# All SMTP services will by default use implicit TLS
postconf smtp_tls_wrappermode=yes

# Explicitly opt-out of implicit TLS (service does not appear to be compatible with `smtp_tls_security_level=encrypt`):
postconf -P smtp-amavis/unix/smtp_tls_wrappermode=no

# Ensure configuration is properly picked up if experimenting in container shell:
supervisorctl restart postfix

Alternatively transport maps can be configured. This adds a new service to master.cf that config/postfix-master.cf is unable to support, making it a bit problematic to support? user-patches.sh should be able to apply it, albeit this approach below may be at risk of appending more than once?

# Add the SMTPS (implicit TLS port 465) service to `master.cf`:
echo "relay-smtps unix -       -       n       -       -       smtp" >> /etc/postfix/master.cf

# Add two overrides, the rest will be inherited from default and `main.cf` `smtp_` parameters:
postconf -P relay-smtps/unix/smtp_tls_security_level=encrypt
postconf -P relay-smtps/unix/smtp_tls_wrappermode=yes

# Configure a transport map, `@destination.test` here will only use this relay when the target address to send to through the relay matches.
# It can be removed to be the default transport for all outbound mail:
echo "destination.test relay-smtps:[mail.relay-service.test]:465" > /etc/postfix/transport
postconf transport_maps=texthash:/etc/postfix/transport

# Ensure configuration is properly applied if using these commands in a container shell:
supervisorctl restart postfix

For now, it makes sense to fix the Amavis service (AFAIK, 127.0.0.1:10025 isn't important here as it's smtpd not smtp, no changes required for it) to not break when enabling implicit outbound TLS.

Later adding support to enable a implicit TLS smtp service seems worthwhile. Then we can leverage transport maps config file to provide a default transport or destination specific. I believe there is a separate relay version of transport maps which allows configuring only for those, which would allow connecting to other MTAs to deliver on port 25 with plain SMTP if desired (opportunistic TLS).

Reproduction example

To troubleshoot this I setup the following environment.

It may be helpful to others beyond the relay issue here when a reproducible environment is needed (easier to identify what's wrong on our end since all config is under control).

Config - MTAs

mail/docker-compose.yml:

services:
  # Your mail-server to receive mail (usually via port 25, which will not be permitted to relay by default).
  # You may submit mail (from your MUA clients authorized via ports 587 & 465 to stmpd) to send / relay to another MTA.
  # Otherwise outbound mail to another MTA must be permitted, such as via SASL auth or allowed networks/subnet,
  # such as a service running within `mta-origin` (allowed), or another container (`mta-origin` must be configured to permit).
  mta-origin:
    image: mailserver/docker-mailserver:11.0
    container_name: dms-origin
    hostname: mail.origin.test
    environment:
      - SSL_TYPE=letsencrypt
      # Setup a single authorized relay service, otherwise use `config/postfix-relayhost_map.cf` for more control:
      - RELAY_HOST=mail.relay-service.test
      - RELAY_PORT=465
      - RELAY_USER=origin-account@relay-service.test
      - RELAY_PASSWORD=relay-password
      # Mostly useful if your MTA doesn't manage receiving mail for the `from`/sender address (eg: `hello@example.test`).
      # The configured RELAY_HOST adds each domain it manages as a sender filter (`main.cf:sender_dependent_relayhost_maps`).
      # Whereas this one sets `main.cf:relayhost` as a fallback for any other sender domain to relay through.
      - DEFAULT_RELAY_HOST=[mail.relay-service.test]:465
      # ENV I am using with my custom entrypoint to automate setup.
      # Our MUA will authorize via this account to submit outbound mail from `mta-origin`:
      - TEST_ACCONT=hello@origin.test
      - TEST_PASS=notsecure
    # This is a minimal example, so no state is persisted:
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./tls:/etc/letsencrypt/live:ro
      - ./entrypoint.sh:/entrypoint.sh:ro
    entrypoint: /entrypoint.sh
    # This example runs completely offline with a custom DNS server and static container IP assigned:
    dns: 172.16.63.128
    networks:
      test_net:
        ipv4_address: 172.16.238.40

  # This would be the relay service MTA (eg: SendGrid), or an external mail provider you use (eg: Gmail).
  # It has an account `origin-account@relay-service.test` to perform submission, like a MUA to `mta-origin`.
  # Same setup as `mta-origin`, except there is no relay ENV config.
  mta-relay:
    image: mailserver/docker-mailserver:11.0
    container_name: dms-relay
    hostname: mail.relay-service.test
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./tls:/etc/letsencrypt/live:ro
      - ./entrypoint.sh:/entrypoint.sh:ro
    environment:
      - SSL_TYPE=letsencrypt
      # `mta-origin` will submit to `mta-relay` via authorization with this account:
      - TEST_ACCONT=origin-account@relay-service.test
      - TEST_PASS=relay-password
    entrypoint: /entrypoint.sh
    dns: 172.16.63.128
    networks:
      test_net:
        ipv4_address: 172.16.238.41

  # The MTA responsible for receiving mail of the `to` address.
  # `mail.relay-service.test` must be authorized in the SPF record of `origin.test`,
  # otherwise `mta-destination` will reject due to SPF fail.
  mta-destination:
    image: mailserver/docker-mailserver:11.0
    container_name: dms-destination
    hostname: mail.destination.test
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./tls:/etc/letsencrypt/live:ro
      - ./entrypoint.sh:/entrypoint.sh:ro
    environment:
        - SSL_TYPE=letsencrypt
        - TEST_ACCONT=hello@destination.test
    entrypoint: /entrypoint.sh
    dns: 172.16.63.128
    networks:
      test_net:
        ipv4_address: 172.16.238.42

# Network config via separate docker-compose
networks:
  test_net:
    external: true
    name: test-ipam

mail/entrypoint.sh:

#!/bin/bash
set -e

function create_testing_account
{
  if [ ! -f "/tmp/docker-mailserver/postfix-accounts.cf" ]
  then
    echo "Initializing test email account..."

    if [ ! -d /tmp/docker-mailserver ]
    then
      mkdir -p /tmp/docker-mailserver
    fi

    local TEST_ACCONT="${TEST_ACCONT:=hello@example.test}"
    local TEST_PASS="${TEST_PASS:=notsecure}"
    setup email add "${TEST_ACCONT}" "${TEST_PASS}"
  fi
}

create_testing_account

# Run the CMD from Dockerfile:
# https://github.com/docker-mailserver/docker-mailserver/blob/0d30b92a83118e7c1cea7cdc40007a3f41d4fe6c/Dockerfile#L301
supervisord --configuration /etc/supervisor/supervisord.conf

Config - DNS

For reference / reproduction, I have also provided the DNS setup. Nothing fancy, just minimal needed for the containers to do their thing and perform SPF checks since unlike our current tests, they are treated as separate servers talking to each other.

Beyond A records to resolve queries, these two are configured for the apex domains:

MX just points to mail.${DOMAIN} (to match hostname/FQDN set in containers, which Postfix configures itself to use, while actual email is exchanged using apex domains (hello@${DOMAIN}).
TXT for SPF record. Adds mx, a:, ip4: entries to authorize when relevant for that zone and the relay tests done.

This is from my earlier offline setup, revised / polished to share here. In another configuration, a local CA service is run with an ACME client to provision certs. That used DNS challenges to provision certs and write the zone files for CoreDNS to read, using zone files was easier to modify DNS records without restarting the whole DNS service.

I opted for step-cli to generate certs instead (as documented in our tests dir), as I was having some issues with the DNS resolution when trying this IPAM docker network handling (in addition to running it in a VM guest network). The certs can then be provided as manual or letsencrypt type for the DMS containers easily enough.

If anyone does know a better self-host DNS solution for this which is easy to grok and perform similar features, that'd be great to know :+1:

dns/docker-compose.yml:

services:
  coredns-mail:
    image: coredns/coredns:1.9.2
    container_name: mail-dns
    # Bound to IP in a VM guest, ThunderBird or other MUA could then connect,
    # when the OS is configured to use this DNS service.
    ports:
      - "172.16.63.128:53:53/udp"
      - "172.16.63.128:53:53"
    volumes:
      - ./Corefile:/Corefile:ro
      - ./zones/:/zones/:ro
    # Probably not relevant, as I don't use this for the DNS address.
    # The split into separate compose files is a separate learning exercise for me :)
    networks:
      test_net:
        ipv4_address: 172.16.238.53

networks:
  test_net:
    name: test-ipam
    ipam:
      driver: default
      config:
        - subnet: "172.16.238.0/24"

dns/Corefile:

example.test {
    file /zones/example.test {
        reload 15s
    }
    log
}

origin.test {
    file /zones/origin.test {
        reload 15s
    }
    log
}

relay-service.test {
    file /zones/relay-service.test {
        reload 15s
    }
    log
}

destination.test {
    file /zones/destination.test {
        reload 15s
    }
    log
}

. {
    #forward . 8.8.8.8 9.9.9.9
    log
}

dns/zones/origin.test:

$ORIGIN origin.test.
$TTL 60

@       SOA     origin.test. admin.origin.test. 2022052400 4h 1h 14d 10m
@       NS      ns1.origin.test.

; Mail
@       MX      10      mail.origin.test.
@       TXT             "v=spf1 mx a:mail.relay-service.test -all"

; A records
@       A       172.16.238.40
mail    A       172.16.238.40
ns1     A       172.16.63.128

dns/zones/relay-service.test:

$ORIGIN relay-service.test.
$TTL 60

@       SOA     relay-service.test. admin.relay-service.test. 2022052400 4h 1h 14d 10m
@       NS      ns1.relay-service.test.

; Mail
@       MX      10      mail.relay-service.test.
@       TXT             "v=spf1 mx -all"

; A records
@       A       172.16.238.41
mail    A       172.16.238.41
ns1     A       172.16.63.128

dns/zones/destination.test:

$ORIGIN destination.test.
$TTL 60

@       SOA     destination.test. admin.destination.test. 2022052400 4h 1h 14d 10m
@       NS      ns1.destination.test.

; Mail
@       MX      10      mail.destination.test.
@       TXT             "v=spf1 mx -all"

; A records
@       A       172.16.238.42
mail    A       172.16.238.42
ns1     A       172.16.63.128

dns/zones/example.test: (an example domain without it's own MTA, but could be accepted by mta-origin and relayed by mta-relay)

$ORIGIN example.test.
$TTL 60

@       SOA     example.test. admin.example.test. 2022052400 4h 1h 14d 10m
@       NS      ns1.example.test.

; Mail
@       TXT             "v=spf1 a:mail.relay-service.test ip4:172.16.238.99 -all"

; `a:mail.relay-service.test` authorizes the mail relay service to send mail on behalf of this domain.
; When the relay service sends the email to it's destination, it will retain the original `from`/sender address,
; while connecting with a `helo` of it's configured hostname FQDN (`mail.relay-service.test`),
; which is what will receive the SPF check on `example.test` when arriving at `destination.test`.

; `ip4:172.16.238.99` belongs to a service sending a test mail on behalf of (`from`/sender field) this domain.
; This is only relevant when sending to a different MTA on port 25, unrelated to relaying which typically 
; requires being authorized via submission ports. For it to be authorized to do so, it is added to the SPF record.

; NOTE: `origin.test` may receive mail with sender address of `example.test`:
; - Via `172.16.238.99` to Port 25. If destined for `origin.test`, it will check SPF here and accept it.
; - Via submission to Port 587 or 465 which requires SASL authorization, thus IP does not require an SPF check.
;   The SPF check is deferred to the MTA that will receive the mail for the intended destination address.
;   `mta-origin` would need to configure an explicit relayhost map for the `example.test` sender, or a default relay,
;    along with SASL authorization configured for that relay if the relay port expects authorization.

; A records
@       A       172.16.238.43
mail    A       172.16.238.43
ns1     A       172.16.63.128

Testing - Client

This is just a small rust program I wrote that uses an unreleased version of lettre to do the bulk of the work. My app smtp-tester just reads a TOML file to configure itself and send a basic email. I remember looking into trying an existing mail client for this but decided this was easier to figure out :sweat_smile:

I wrapped it into a container which was useful during testing (originally to debug a previous issue we got confused about with the postfix policy service that checks dovecot quota). Nothing particularly special about it, but if someone is interested in it, I can look at tidying up the code to publish it, eventually I'd like to use it in our test suite instead of netcat/nc.

docker run --rm -t --name mail-test \
  --dns 172.16.63.128 \
  --network test-ipam \
  --ip 172.16.238.99 \
  -v "${PWD}/data/:/data/:ro" \
  smtp-tester --file /data/examples/relay.toml

mua/data/examples/relay.toml:

# Test email:
##########

from    = "hello@example.test"
to      = "hello@destination.test"
subject = "Testing relay"
body    = "Message content"

# Connection details:
#################

# Setting MTA host, port, TLS connection type and login to authorize submission
# Can use port 587 with 'Explicit' type too, does not affect what relay host was configured for
host  = "origin.test"
port  = 465
tls   = "Implicit"
creds = { user = "hello@origin.test", pass = "notsecure" }

# Root CA cert to validate against the tls cert provided from the MTA
# Usually this is not required as it's in your systems trust store
ca_cert = "/data/tls/ca-cert.pem"

I've also tested Thunderbird via GUI connecting to origin.test and destination.test MTAs to send/receive through the relay-service.test MTA. That went fine :+1:

@georglauterbach

@polarathene wrote the new and much improved wrappers I believe.

I mostly extracted sections of code into smaller scripts, especially with relay code as it was being managed in two separate files and falling out of sync.

I didn't get around to grokking it well enough to consider refactoring further, but I could definitely look into improving that now :sunglasses:

Can you please have a look @polarathene and see whether we can make the relay host setup more uniform? We may be able to use sed for Postfix's master.cf?

For now, master.cf can just add an explicit -o smtp_tls_wrappermode=no to smtp-amavis. No need for sed?

We'd still need main.cf to enable smtp_tls_wrappermode=yes, but only if configuring outbound SMTP for implicit TLS with port 465. I'm not wanting to add an ENV for that, nor too keen on the relay script trying to detect if it should toggle it via port (which in most cases may be ok). It seems it might be better to just document adding a line to config/postfix-main.cf to enable that parameter?

Alternatively, if we support an implicit TLS SMTP service in master.cf, I think the current config (ENV or postfix-relaymap.cf) allows for prefixing it to the relay host which might work (I have not tested such), although that config file is not monitored for changes, so any updates while the container is up requires restarting postfix within the container or other means.

We currently use smtp_tls_security_level = may as the default in main.cf and elsewhere. Why don't we use enforce here?

SMTP is required to use opportunistic TLS (may), at least on the receiving end with port 25. I think an RFC mandates it for delivery and Postfix docs explicitly discourage enforcing TLS for SMTPD / inbound port 25.

For outbound mail, if you're not concerned about possibly sending to some mail server that for whatever reason doesn't support secure connections, then I guess it's fine?

IIRC, MTAs may support TLS, but not a secure cipher that can be negotiated... at which point delivery fails with no fallback to unencrypted delivery. Our STARTTLS ports other than port 25 all require successfully negotiating a secure connection.

I assume we require STARTTLS (I mean encryption with STARTTLS) anyway, even when starting on port 25, right? Were does this happen

grep smtpd_ /etc/postfix/main.cf (inbound SMTP) and grep smtp_ /etc/postfix/main.cf (outbound SMTP) is the related configuration we set (beyond Postfix defaults). For STARTTLS that's covered by stmpd?_tls_security_level = may by both, and would apply to all ports that support SMTP protocol I think.

Other settings for stmpd?_ parameters cover enabling auth or restricting ciphers to allow negotiating with, there's also some similar parameters specific to relays that can be supported too, if your mail server does more than relay through another MTA.

In master.cf we have additional services/ports enabled for smtpd/inbound, which is submission (grep submission /etc/services will show it maps to port 587) and smtps (port 465 with legacy service name, it's also in /etc/services mapping as an alias to the present service name submissions). These two service definitions in master.cf set parameter overrides, such as enforcing authorization/login, and requiring an encrypted connection to use. smtps additionally sets smtpd_tls_wrappermode=yes to configure as a typical TLS port/service, without requiring STARTTLS connection upgrade like port 587 does.

Some relay services may support other ports to connect on (2525, 25) but enforce authorization (and I believe encryption?). Some may offer 465 with explicit (STARTTLS) instead of implicit TLS. I don't know how common that is now though. So it's all flexible in config as described above :)

Wow, this was a lot of effort 👍🏼 So, basically, we can just adjust master.cf?

And thank your for answering my question :)

So, basically, we can just adjust master.cf?

Adding -o smtp_tls_wrappermode=no to smtp-amavis service is needed.
User then explicitly configures themselves, or scripts/helpers/relay.sh must support detecting implicit TLS and either:
- Enforce implicit TLS for all outbound SMTP via main.cf: postconf smtp_tls_wrappermode=yes
- Configure a separate smtp service in master.cf for relay-hosts and set in ~~main.cf:relay_transport~~ (_EDIT: Misunderstanding of config relay_transport, that is the wrong choice, our relay support would configure default_transport or `sender dependent default transports_).

I'll contribute a PR this week.

docker-mailserver / docker-mailserver