Transport security in SMTP world is merely Opportunistic, there is no practice like in the Web to always require encryption with matching DNS-ID and trusted certificate. Additionally, things get complicated with the use of MX records to delegate mail handling.
Overall, here are possible options for each SMTP connection, sorted by protection "strength":
No encryption (no protection altogether).
TLS without authentication e.g. self-signed cert. (protects against passive attacks only, server key can be spoofed)
TLS with server cert. authentication (protects against passive attacks, MX records can be spoofed)
TLS with server cert. authentication and DNSSEC (protects against passive attacks, STARTTLS support can be spoofed to downgrade connection to plaintext)
TLS with server cert. authentication and MTA-STS (protects against passive and active attacks, but policy expiry makes it unreliable)
TLS with server cert. authentication and DNSSEC+DANE (protects against active and passive attacks, best option)
Each of these options is better than the previous one so it makes sense to support all of them without directly falling back to the weakest one (no encryption).
Interaction with REQUIRETLS
SMTP REQUIRETLS extension requires TLS with valid and matching certificates and MX records authentication using either DNSSEC or MTA-STS. So messages using it are handled in a "strongest security or fail delivery" way instead of Opportunistic Security approach typically employed by MTAs.
Though, MTA-STS and DANE have similar but not same security characteristics (as was noted in the draft spec review, see References). This might degrade the actual security provided by extension in the long term if DANE becomes more widespread than MTA-STS (unlikely).
Proposal
Implement REQUIRETLS extension (issue #123)
Provide policy options to require certain security level for all messages (already done, see authenticate_mx, require_mx_auth and require_tls directives for remote module).
(Possibly) Implement downgrade protection for non-REQUIRETLS messages by remembering the authentication/encryption status like chasquid does it.
It will help with messages sent without using REQUIRETLS extension (as practice shows, SMTP extensions support is poor and grows slowly). Though, it is not clear whether to group options by protection scope (passive/active attacks) for simplicity or provide high granularity. Later option may be better for security but also increases the chance of unexpected interactions.
Matching TLSA record with DANE-TA (Usage 2) or DANE-EE (Usage 3) can replace PKIX and ServerName checks and provides the essentially same (or even stronger) protection against active attacks.
Observation: Most deployed TLSA records use DANE-EE.
Transport security in SMTP world is merely Opportunistic, there is no practice like in the Web to always require encryption with matching DNS-ID and trusted certificate. Additionally, things get complicated with the use of MX records to delegate mail handling.
Overall, here are possible options for each SMTP connection, sorted by protection "strength":
Each of these options is better than the previous one so it makes sense to support all of them without directly falling back to the weakest one (no encryption).
Interaction with REQUIRETLS
SMTP REQUIRETLS extension requires TLS with valid and matching certificates and MX records authentication using either DNSSEC or MTA-STS. So messages using it are handled in a "strongest security or fail delivery" way instead of Opportunistic Security approach typically employed by MTAs.
Though, MTA-STS and DANE have similar but not same security characteristics (as was noted in the draft spec review, see References). This might degrade the actual security provided by extension in the long term if DANE becomes more widespread than MTA-STS (unlikely).
Proposal
Implement REQUIRETLS extension (issue #123)
Provide policy options to require certain security level for all messages (already done, see authenticate_mx, require_mx_auth and require_tls directives for remote module).
(Possibly) Implement downgrade protection for non-REQUIRETLS messages by remembering the authentication/encryption status like chasquid does it. It will help with messages sent without using REQUIRETLS extension (as practice shows, SMTP extensions support is poor and grows slowly). Though, it is not clear whether to group options by protection scope (passive/active attacks) for simplicity or provide high granularity. Later option may be better for security but also increases the chance of unexpected interactions.
References