Forum is dropping emails

[kousu]

For the last few weeks we've been intermittently missing forum notifications from https://forum.spinalcordmri.org. Some people will receive them, some won't.

@alexfoias discovered it was a problem with spam filters suddenly getting picky about our forum emails. @joshuacwnewton discovered the problem is that SendGrid, the SMTP-as-a-Service that we're using, has been under attack? pressure? I don't know what you'd call it. Anyway here's the story: https://krebsonsecurity.com/2020/08/sendgrid-under-siege-from-hacked-accounts/. As a result, several (but not all!) of their free-tier servers are now on spamlists, e.g. https://talosintelligence.com/reputation_center/lookup?search=167.89.100.223 or https://www.spamhaus.org/sbl/query/SBL499109. This explains why the problem is intermittent.

[kousu]

We could fix this by:

1. Paying the paid tier of SendGrid; those servers seem to be unaffected.
2. Switching to a different SMTP service (Discourse has some recommendations): MailGun, MailJet, Elastic. There's also Amazon SES. I'm sure there's others.
3. Running our own mail server, where we manage our own reputation.

I am in favour of option 3; the whole point of using SaaS is to make monitoring and reliability and, in this case, managing reputation into someone else's problem that you don't need to worry about. But clearly in this case SendGrid wasn't able to achieve that. So, since now we have to worry about reputation anyway, we might as well take charge of it instead of outsourcing.

I'm implementing option 3 now on the server using OpenSMTPD. I'll document my work here, and then try to summarize it in the main README.

This will supersede #13.

[kousu]

The first step (after ssh forum.spinalcordmri.org of course) was to install it and bootstrap it with a basic config:

sudo apt-get install opensmtpd
cat > /etc/smtpd.conf <<EOF
listen on eth0
table aliases file:/etc/aliases
accept from any for domain "forum.spinalcordmri.org" alias <aliases> deliver to mbox
accept for local alias <aliases> deliver to mbox
accept for any relay
EOF
sudo systemctl enable --now opensmtpd

To check on its status, use

journalctl -u opensmtpd -f

:+1:

[kousu]

It ran for a couple minutes, but then partially crashed and wouldn't come back up.

I dug into it and discovered this known bug: https://bugs.launchpad.net/ubuntu/+source/opensmtpd/+bug/1840586, the person who wrapped it in systemd for Linux didn't check their work I guess. I manually backported the fix; but this is only because that server is running 18.04-LTS; it would be equally good to just reinstall the server on an updated Ubuntu -- this shouldn't be a huge deal, we just need to backup and restore the forum, and it's all containerized so that should be safe, but I'll leave that alone for now.

Anyway so the fix was to vi /lib/systemd/system/opensmtpd.service and apply this patch, by hand:

--- /lib/systemd/system/opensmtpd.service.old   2020-11-05 01:20:51.164473166 +0000
+++ /lib/systemd/system/opensmtpd.service   2020-11-03 21:22:34.309085523 +0000
@@ -6,7 +6,8 @@
 [Service]
 Type=forking
 ExecStart=/usr/sbin/smtpd
-ExecStop=/usr/sbin/smtpctl stop # backported fix for https://bugs.launchpad.net/ubuntu/+source/opensmtpd/+bug/1840586
+ExecStop=/bin/kill -15 $MAINPID

 [Install]
 WantedBy=multi-user.target

Then I just pkill smtpd && systemctl restart opensmtpd and it came back stably.

[kousu]

To test if it's working, I sent mails from the command line with

echo "Test Message" | mail -s "This is a message" me@example.org

Watching journalctl -f -u opensmtpd in a separate tab while doing this is helpful to confirm things are alive and working, as is trying different target addresses; but sending to @gmail.com addresses was a mistake, it pre-poisoned their filters against us (but the damage doesn't seem to be permanent). Better was to target my own mail server where I was able to log in and watch the the mail log, so I could see both sending and receiving in real time.

A very helpful site to target is https://www.mail-tester.com/. You send an email to an address it gives you and it tells you how to fix up your spamminess reputation. But it only gives 3 free tests a day, so it's important to use them wisely.

[kousu]

Next, with @alexfoias, I set up a proper email server identity. That is, the identity -- for reputation purposes -- of the server itself. Vouching for your server is important to get accepted as legitimate email.

1. Make sure /etc/hostname and /etc/mailname are both "forum.spinalcordmri.org"; reboot the server to make sure this takes.
2. Make sure the DNS MX record is correct: log in to namecheap, scroll to the email section, switch it to "Custom MX" and write in MX forum = forum.spinalcordmri.org, priority 0.
   • to test: dig MX forum.spinalcordmri.org should be "forum.spinalcordmri.org"
3. Set up SPF: again in namecheap, in the main records section, add TXT forum. = "v=spf1 a mx ip4:159.89.119.65 ~all"
   • to test: dig TXT forum.spinalcordmri.org should return the string above.
4. Reverse DNS: log in to the Droplet control panel at DigitalOcean (DO) and make the reverse DNS "forum.spinalcordmri.org" -- on DO this is done by changing the name of the Droplet: https://www.digitalocean.com/community/questions/how-do-i-set-up-reverse-dns-for-my-ip.
   • to test: dig +short -x $(dig +short forum.spinalcordmri.org) should return "forum.spinalcordmri.org".
5. DMARC: again in namecheap, add a record TXT _dmarc.forum. = "v=DMARC1; p=none"; but I'm not sure this achieves anything really.
   • to test: dig TXT _dmarc.forum.spinalcordmri.org should return the string above.

If any of the DNS tests are failing after turned, it might be DNS caching, so try a different server with dig's @ syntax, e.g. dig @4.2.2.2 +short -x $(dig +short forum.spinalcordmri.org)

During this, I discovered an inconsistency. Previously, there had been records set up, but they had been for spinalcordmri.org, not for forum.spinalcordmri.org; however the Discourse forum sends as noreply@forum.spinalcordmri.org.

I skipped DKIM. I'm unconvinced it's that good and it's pretty complicated (== fragile). But maybe we can do it later if we have more problems.

I tested a couple of times using https://mail-tester.com to see how these improvements were being received.

[kousu]

Next, Discourse, unlike more traditional webapps, cannot just send mail using mail, it insists on having SMTP credentials (probably because it's all Containerized and stuff :roll_eyes: ), which means we need to set up a user and authentication on the server so it can log in to itself. So we need to set up outgoing mails via SMTP, and not just via local senders.

1. useradd -s /usr/sbin/nologin forum && passwd forum
   • run a password generator (e.g. xkcdpass) to generate the password; keep a tempoary note of it.
2. find the SSL cert on the server; this is a prereq for SMTP auth. As of this writing, we're using the Discourse-generated cert, and it is stored in /var/discourse/shared/standalone/ssl/
3. reconfigure /etc/smtpd.conf to add TLS and authentication; also add tls verify to insist that outgoing mails also travel over TLS (which is a reasonable assumption these days); if that turns out to be too strong we can reduce it to tls which is just ((note: in the most recent opensmtpd, the one that's probably in ubuntu 20.04, it's changed: tls verify became tls and tls became tls no-verify, which just underlines that it's a reasonable practice to force TLS as much as possible)):

# /etc/smtpd.conf

pki forum.spinalcordmri.org certificate "/var/discourse/shared/standalone/ssl/forum.spinalcordmri.org.cer"
pki forum.spinalcordmri.org key "/var/discourse/shared/standalone/ssl/forum.spinalcordmri.org.key"

listen on eth0 tls-require pki forum.spinalcordmri.org auth
table aliases file:/etc/aliases
accept from any for domain "forum.spinalcordmri.org" alias <aliases> deliver to mbox
accept for local alias <aliases> deliver to mbox
accept for any relay hostname "forum.spinalcordmri.org" tls verify

To test I need to send from a remote server. I considered using a mail client like Thunderbird, but that got difficult because they all wanted dual SMTP/IMAP accounts and I haven't set up IMAP. I considered python.smtplib but before I started down that road I googled and found swaks. So, to test, I installed that and from a different machine, I did:

$ swaks --to me@example.com --from noreply@forum.spinalcordmri.org --server forum.spinalcordmri.org -p 25 --auth-user forum --tls-verify --tls
Password: xxxxxxxxxxxxxxxxxxxxxxx
=== Trying forum.spinalcordmri.org:25...
=== Connected to forum.spinalcordmri.org.
<-  220 forum.spinalcordmri.org ESMTP OpenSMTPD
 -> EHLO requiem
<-  250-forum.spinalcordmri.org Hello laptop [x.x.x.x], pleased to meet you
<-  250-8BITMIME
<-  250-ENHANCEDSTATUSCODES
<-  250-SIZE 36700160
<-  250-DSN
<-  250-STARTTLS
<-  250 HELP
 -> STARTTLS
<-  220 2.0.0: Ready to start TLS
=== TLS started with cipher TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256
=== TLS no local certificate set
=== TLS peer DN="/CN=forum.spinalcordmri.org"
 ~> EHLO requiem
<~  250-forum.spinalcordmri.org Hello requiem [172.98.71.42], pleased to meet you
<~  250-8BITMIME
<~  250-ENHANCEDSTATUSCODES
<~  250-SIZE 36700160
<~  250-DSN
<~  250-AUTH PLAIN LOGIN
<~  250 HELP
 ~> AUTH LOGIN
<~  334 xxxxxxxxxxxxx
 ~> xxxxxxxxxx=
<~  334 xxxxxxxxxxx
 ~> xxxxxxxxxxxxx=
<~  235 2.0.0: Authentication succeeded
 ~> MAIL FROM:<noreply@forum.spinalcordmri.org>
<~  250 2.0.0: Ok
 ~> RCPT TO:<me@example.com>
<~  250 2.1.5 Destination address valid: Recipient ok
 ~> DATA
<~  354 Enter mail, end with "." on a line by itself
 ~> Date: Wed, 04 Nov 2020 20:07:09 -0500
 ~> To: me@example.com
 ~> From: noreply@forum.spinalcordmri.org
 ~> Subject: test Wed, 04 Nov 2020 20:07:09 -0500
 ~> Message-Id: <20201104200709.079426@requiem>
 ~> X-Mailer: swaks vDEVRELEASE jetmore.org/john/code/swaks/
 ~> 
 ~> This is a test mailing
 ~> 
 ~> 
 ~> .
<~  250 2.0.0: 8ccb62c7 Message accepted for delivery
 ~> QUIT
<~  221 2.0.0: Bye
=== Connection closed with remote host.

I checked my email and indeed, I'd gotten "This is a test mailing" from it, so that looks like it's working :)

[kousu]

I took a detour to think about incoming email at the same time. Discourse supports reply-by-email so for that we need the mail server to handle incoming email, and somehow be able to get it over into Discourse.

It turns out there are two methods:

1. n00b method: set up a POP3 account somewhere: https://meta.discourse.org/t/set-up-reply-via-email-support/14003
2. instant-replies method: turn on their built-in containerized mail server: https://meta.discourse.org/t/straightforward-direct-delivery-incoming-mail/49487

Neither of these options are great. I can set up POP3 (by installing dovecot, I know dovecot) but there will be a polling delay so replies won't show up immediately. But using their built in mail-server will conflict, and apparently it is literally incoming-only, so I couldn't even make it handle sending out notifications; also it requires mucking about in the Discourse admin GUI to pull out an API key which is just like...why.

I think I will leave it alone for now. If we decide we want this, we can add dovecot to the server. It won't be the end of the world to have an extra daemon and a little bit of delay.

--

I decided, with that in mind, to disable incoming mail, for now, since it won't go anywhere useful (there's also a recipient option which is a weaker form: defining an allowlist of acceptable addresses and dropping all others). I also switched to maildir because mbox is inefficient!

# /etc/smtpd.conf
pki forum.spinalcordmri.org certificate "/var/discourse/shared/standalone/ssl/forum.spinalcordmri.org.cer"
pki forum.spinalcordmri.org key "/var/discourse/shared/standalone/ssl/forum.spinalcordmri.org.key"

listen on eth0 tls-require pki forum.spinalcordmri.org auth-optional
table aliases file:/etc/aliases
# incoming remote mail
#accept from any for domain "forum.spinalcordmri.org" alias <aliases> deliver to maildir "~/.mail"
# incoming localhost mail
accept for local alias <aliases> deliver to maildir "~/.mail"

# outgoing mail
accept for any relay hostname "forum.spinalcordmri.org" tls verify

[kousu]

Because opensmtpd is relying on letsencrypt certs generated via Discourse, which is doing it on a schedule in a container unconnected from the rest of the system, there's no good way to trigger opensmtpd's reloading of certs. To glue it together, add this to root's crontab:

# daily reboot of opensmtpd, so it can pick up new certs generated inside of Discourse
30 2 * * *  systemctl restart opensmtpd

[kousu]

I've found a related glitch that might be causing problems. smtp.polymtl.ca is configured as a forwarder for certain accounts, but this breaks SPF, and indeed Gmail recognizes this and complains about it.

Sent directly:

swaks --to xxxxxxxx@gmail.com --from noreply@forum.spinalcordmri.org --server forum.spinalcordmri.org -p 25 --auth-user forum --tls-verify --tls

but sent via the forwarder (which goes to my same gmail address):

swaks --to xxxxxxx@polymtl.ca --from noreply@forum.spinalcordmri.org --server forum.spinalcordmri.org -p 25 --auth-user forum --tls-verify --tls

There's nothing I can do about this. Maybe TI at poly can look into this workaround? It seems to be...okay, at least for gmail, because once you 'unspam' a couple of messages sent this way the filter learns to ignore the SPF fail.

[alexfoias]

Wouldn't that be the case for all the email adresses (not only polymtl) that have a forward to another email ?

[kousu]

Wouldn't that be the case for all the email adresses (not only polymtl) that have a forward to another email ?

Yes, totally. The people who invented SPF even list it as a known bug. I guess it's a minor enough issue that no one really worries about it too much. I'm probably the only one on this forum that even has polymtl forwarding set up, and that's not even the address I'm using on the forum.

[kousu]

One last tweak: Discourse used submission on port 587 instead of smtp on port 25 by default, so just to play nice I enabled that in /etc/smtpd.conf:

pki forum.spinalcordmri.org certificate "/var/discourse/shared/standalone/ssl/forum.spinalcordmri.org.cer"
pki forum.spinalcordmri.org key "/var/discourse/shared/standalone/ssl/forum.spinalcordmri.org.key"

listen on eth0 tls-require pki forum.spinalcordmri.org auth-optional
listen on eth0 tls-require pki forum.spinalcordmri.org auth port 587
table aliases file:/etc/aliases
accept from any for domain "forum.spinalcordmri.org" alias <aliases> deliver to mbox
accept for local alias <aliases> deliver to mbox
accept for any relay hostname "forum.spinalcordmri.org" tls verify

I re-ran the discourse install script as instructed to change the outgoing email settings:

root@forum:/var/discourse# ./discourse-setup 
The configuration file containers/app.yml already exists!

. . . reconfiguring . . .

Saving old file as app.yml.2020-11-05-193026.bak
Stopping existing container in 5 seconds or Control-C to cancel.
+ /usr/bin/docker stop -t 10 app
app

Found 1GB of memory and 1 physical CPU cores
setting db_shared_buffers = 128MB
setting UNICORN_WORKERS = 2
containers/app.yml memory parameters updated.

Hostname for your Discourse? [forum.spinalcordmri.org]: 

Checking your domain name . . .
Connection to forum.spinalcordmri.org succeeded.
Email address for admin account(s)? [spinalcordmri@gmail.com]: 
SMTP server address? [smtp.sendgrid.net]: forum.spinalcordmri.org
SMTP port? [587]: 587
SMTP user name? [xxxxxx]: xxxxxx
SMTP password? [xxxxxxxxxxxxxx]: xxxxxxxxxxx
Optional email address for Let's Encrypt warnings? (Enter 'OFF' to disable.) [spinalcordmri@gmail.com]: 

Does this look right?

Hostname      : forum.spinalcordmri.org
Email         : spinalcordmri@gmail.com
SMTP address  : forum.spinalcordmri.org
SMTP port     : 587
SMTP username : xxxxxxxx
SMTP password : xxxxxxxxxxx
Let's Encrypt : spinalcordmri@gmail.com

ENTER to continue, 'n' to try again, Ctrl+C to exit: 

It took a while but it came back up just fine.

[kousu]

incoming

Decision from ye grande admin @jcohenadad :

hey guys, i haven’t followed the thread closely but i’m not a big fan of email reply

So we're just going to leave reply-by-email off.

---

Since we're not accepting any incoming mail, to reduce the attack-surface, I just turned off incoming completely. The config is now:

pki forum.spinalcordmri.org certificate "/var/discourse/shared/standalone/ssl/forum.spinalcordmri.org.cer"
pki forum.spinalcordmri.org key "/var/discourse/shared/standalone/ssl/forum.spinalcordmri.org.key"

listen on eth0 tls-require pki forum.spinalcordmri.org auth-optional
listen on eth0 tls-require pki forum.spinalcordmri.org auth port 587
table aliases file:/etc/aliases
#accept from any for domain "forum.spinalcordmri.org" alias <aliases> deliver to mbox
accept for local alias <aliases> deliver to mbox
accept for any relay hostname "forum.spinalcordmri.org" tls verify

[kousu]

Gmail and Hotmail/Outlook seem to be accepting our mail. My personal server is accepting it, as is my academic institution.

But in /var/log/mail.log so far I've found:

Nov  5 19:55:07 forum smtpd[29392]: 9869d9ca558009ec mta event=connecting address=tls://67.231.152.210:25 host=mx0b-0026b901.pphosted.com
Nov  5 19:55:07 forum smtpd[29392]: 9869d9ca558009ec mta event=connected
Nov  5 19:55:07 forum smtpd[29392]: 9869d9ca558009ec mta event=error reason=BANNER rejected: 554 Blocked - see https://ipcheck.proofpoint.com/?ip=159.89.119.65
Nov  5 19:55:07 forum smtpd[29392]: smtp-out: Disabling route [] <-> 67.231.152.210 (mx0b-0026b901.pphosted.com) for 15s
Nov  5 19:55:09 forum smtpd[29392]: 9869d9cbb3c188a2 mta event=connecting address=tls://67.231.144.201:25 host=mx0a-0026b901.pphosted.com
Nov  5 19:55:09 forum smtpd[29392]: 9869d9cbb3c188a2 mta event=connected
Nov  5 19:55:10 forum smtpd[29392]: 9869d9cbb3c188a2 mta event=error reason=BANNER rejected: 554 Blocked - see https://ipcheck.proofpoint.com/?ip=159.89.119.65

I went to that URL and submitted a false positive report.

According to https://mail-tester.com our IP 159.89.119.65 is not on any of the major blocklists. ProofPoint must have their own blocklist, though, or perhaps they are just greylisting? There will probably be the odd destination that we need to check off one by one.

[kousu]

The spambots are extremely quick. We also got this in the log:

Nov  5 20:05:31 forum smtpd[29392]: 9869d9cc88260339 smtp event=failed-command address=148.163.128.145 host=148.163.128.145 command="mail from: <7wqockg@sw4.24h.com.vn>" result="530 5.5.1 Invalid command: Must issue a STARTTLS command first"
Nov  5 20:05:31 forum smtpd[29392]: 9869d9cc88260339 smtp event=failed-command address=148.163.128.145 host=148.163.128.145 command="rcpt to: <postmaster@spinalcordmri.org>" result="503 5.5.1 Invalid command: Command not allowed at this point."
Nov  5 20:05:31 forum smtpd[29392]: 9869d9cc88260339 smtp event=failed-command address=148.163.128.145 host=148.163.128.145 command="rcpt to: <tg44g@onlylady.com>" result="503 5.5.1 Invalid command: Command not allowed at this point."
Nov  5 20:05:31 forum smtpd[29392]: 9869d9cc88260339 smtp event=failed-command address=148.163.128.145 host=148.163.128.145 command="rcpt to: <nfwh@oq4.rayli.com.cn>" result="503 5.5.1 Invalid command: Command not allowed at this point."

however, because I insisted on tls-required, they are foiled. It wouldn't be hard for them to use TLS, but spambots are cheap and quick and don't try very hard. And anyway, incoming mail is completely disabled.

[kousu]

Before I close this, I'm going to update README.md with the additional instructions in case we need to rebuild the server.

[kousu]

Received this in the maillog:

Nov 13 22:52:32 forum smtpd[22565]: 84adcdcd7f8a63f1 mta event=error reason=EHLO rejected: 550-REJECT: 159.89.119.65 is in csi.cloudmark.com :Listed as Poor Reputation Sender. Cloudmark Sender Intelligence ReputationRemediation Portal https://csi.cloudmark.com/en/reset (ID:550:3:0)

I submitted a report to the URL listed, so hopefully they unblock us soon.

[kousu]

Received this in the maillog:

Nov 13 22:52:32 forum smtpd[22565]: 84adcdcd7f8a63f1 mta event=error reason=EHLO rejected: 550-REJECT: 159.89.119.65 is in csi.cloudmark.com :Listed as Poor Reputation Sender. Cloudmark Sender Intelligence ReputationRemediation Portal https://csi.cloudmark.com/en/reset (ID:550:3:0)

I submitted a report to the URL listed, so hopefully they unblock us soon.

They've emailed me back to say they've unblocked us.

[joshuacwnewton]

Thanks much for taking care of that, @kousu!

[kousu]

We're having problems with hotmail.com (weirdly, outlook.com seems fine?):

Nov 17 17:53:11 forum smtpd[26729]: 7cc0da84952a4198 mta event=delivery evpid=78eddf730209220a from=<noreply@forum.spinalcordmri.org> to=<______________@hotmail.com> rcpt=<-> source="159.89.119.65" rel
ay="104.47.58.33 (104.47.58.33)" delay=1s result="PermFail" stat="550 5.7.1 Unfortunately, messages from [159.89.119.65] weren't sent. Please contact your Internet service provider since part of
 their network is on our block list (S3140). You can also refer your provider to http://mail.live.com/mail/troubleshooting.aspx#errors. [DM6NAM10FT038.eop-nam10.prod.protection.outlook.com]"
Nov 17 17:53:11 forum smtpd[26729]: 7cc0da84952a4198 mta event=error reason=Connection closed unexpectedly

I'm going to hope this clears up by itself. https://postmaster.live.com/pm/troubleshooting.aspx says

IPs not previously used to send email typically don't have any reputation built up in our systems. As a result, emails from new IPs are more likely to experience deliverability issues. Once the IP has built a reputation for not sending spam, Outlook.com will typically allow for a better email delivery experience.

though the error message makes it sound like it might be a more specific problem with DigitalOcean. So, I don't know.

[kousu]

Ah, actually I am filling out the Microsoft sender report form at http://go.microsoft.com/fwlink/?LinkID=614866. According to ye olde randome blogge poste, and also this, this can get our IP allowlisted even if the subnet around us is blocked. But it will take a few days and some nagging.

Outlook Support request number: SR1512866220

[kousu]

Nothing that I've seen from Microsoft yet :1st_place_medal:

[kousu]

I observed some more delivery errors due to tls require, for example:

Nov 10 12:11:50 forum smtpd[12634]: 956edc9dc74889f2 smtp event=connected address=172.17.0.2 host=172.17.0.2
Nov 10 12:11:50 forum smtpd[12634]: 956edc9dc74889f2 smtp event=starttls address=172.17.0.2 host=172.17.0.2 ciphers="version=TLSv1.2, cipher=ECDHE-RSA-AES256-GCM-SHA384, bits=256"
Nov 10 12:11:50 forum smtpd[12634]: 956edc9dc74889f2 smtp event=authentication user=forum address=172.17.0.2 host=172.17.0.2 result=ok
Nov 10 12:11:50 forum smtpd[12634]: 956edc9dc74889f2 smtp event=message address=172.17.0.2 host=172.17.0.2 msgid=3b5e2145 from=<noreply@forum.spinalcordmri.org> to=<[redacted]@sina.com> size=50034 
ndest=1 proto=ESMTP
Nov 10 12:11:50 forum smtpd[12634]: 956edc9dc74889f2 smtp event=closed address=172.17.0.2 host=172.17.0.2 reason=quit
Nov 10 12:11:51 forum smtpd[12634]: 956edca09d1446f6 mta event=connecting address=tls://123.126.45.192:25 host=123.126.45.192
Nov 10 12:11:51 forum smtpd[12634]: 956edca09d1446f6 mta event=connected
Nov 10 12:11:55 forum smtpd[12634]: 956edca2ec199051 mta event=connecting address=tls://121.14.32.117:25 host=121.14.32.117
Nov 10 12:11:55 forum smtpd[12634]: 956edca2ec199051 mta event=connected
Nov 10 12:11:57 forum smtpd[12634]: 956edca1c4c8e3fc mta event=connecting address=tls://39.156.6.104:25 host=39.156.6.104
Nov 10 12:11:57 forum smtpd[12634]: 956edca1c4c8e3fc mta event=connected
Nov 10 12:12:00 forum smtpd[12634]: 956edca2ec199051 mta event=error reason=TLS required but not supported by remote host
Nov 10 12:12:00 forum smtpd[12634]: smtp-out: Disabling route [] <-> 121.14.32.117 (121.14.32.117) for 15s
Nov 10 12:12:05 forum smtpd[12634]: 956edca1c4c8e3fc mta event=error reason=TLS required but not supported by remote host
Nov 10 12:12:05 forum smtpd[12634]: smtp-out: Disabling route [] <-> 39.156.6.104 (39.156.6.104) for 15s
Nov 10 12:12:07 forum smtpd[12634]: 956edca09d1446f6 mta event=error reason=TLS required but not supported by remote host
Nov 10 12:12:07 forum smtpd[12634]: smtp-out: Disabling route [] <-> 123.126.45.192 (123.126.45.192) for 15s
Nov 10 12:12:07 forum smtpd[12634]: smtp-out: No valid route for [connector:[]->[relay:sina.com,starttls,heloname=forum.spinalcordmri.org],0x0]

or

Nov 10 17:48:04 forum smtpd[12634]: 956edd2663b3e1e3 smtp event=connected address=172.17.0.2 host=172.17.0.2
Nov 10 17:48:04 forum smtpd[12634]: 956edd1a3aeede2a mta event=starttls ciphers=version=TLSv1.2, cipher=ECDHE-RSA-AES256-GCM-SHA384, bits=256
Nov 10 17:48:04 forum smtpd[12634]: 956edd1a3aeede2a mta event=error reason=SSL certificate check failed
Nov 10 17:48:04 forum smtpd[12634]: smtp-out: Disabling route [] <-> 162.105.129.21 (mx1.pku.edu.cn) for 15s
Nov 10 17:48:04 forum smtpd[12634]: 956edd0f1f16f462 mta event=starttls ciphers=version=TLSv1.2, cipher=ECDHE-RSA-AES256-GCM-SHA384, bits=256
Nov 10 17:48:04 forum smtpd[12634]: 956edd0f1f16f462 mta event=error reason=SSL certificate check failed
Nov 10 17:48:04 forum smtpd[12634]: smtp-out: Disabling route [] <-> 162.105.129.22 (mx2.pku.edu.cn) for 15s
Nov 10 17:48:04 forum smtpd[12634]: smtp-out: No valid route for [connector:[]->[relay:pku.edu.cn,starttls,heloname=forum.spinalcordmri.org],0x0]

So I removed that from the config. The config is now:

root@forum:~# cat /etc/smtpd.conf 

pki forum.spinalcordmri.org certificate "/var/discourse/shared/standalone/ssl/forum.spinalcordmri.org.cer"
pki forum.spinalcordmri.org key "/var/discourse/shared/standalone/ssl/forum.spinalcordmri.org.key"

listen on eth0 tls-require pki forum.spinalcordmri.org auth-optional
listen on eth0 tls-require pki forum.spinalcordmri.org auth port 587
table aliases file:/etc/aliases
#accept from any for domain "forum.spinalcordmri.org" alias <aliases> deliver to maildir "~/.mail" 
accept for local alias <aliases> deliver to maildir "~/.mail" 
accept for any relay hostname "forum.spinalcordmri.org"

Until recently OpenSMTPd suggested avoiding this exactly for this reason:

Note that the tls and tls verify options should only be used in private networks as they will prevent proper relaying on the Internet.

opensmtpd still tries to make a TLS connection if it can, but it isn't mandatory and therefore can be disabled by someone fairly determined. The newer OpenSMTPd changed suppressed that advice, but I guess not everyone is prepared for the cyberpunk future yet. Anyway, not a huge deal for us since most of the forum messages are already public.

[kousu]

I think this is done for now. I'll just leave one last note that it is a really good idea to check in every so often with what's in:

ssh forum.spinalcordmri.org grep event=error /var/log/mail.log