remote smtp server does not implement smtputf8 extension, required by message

[lmeunier]

mox fails to send mails to some SMTP servers with the following error:

mox[6651]: l=error m="storing delivery error" pkg=queue deliveryerror="remote smtp server does not implement smtputf8 extension, required by message, transient, transient" cid=18e66f38ee7 from=robot@infra.example.com attempts=0 cid=18e66f38eec permanent=false code=0 secode=
mox[6651]: l=error m="temporary failure delivering from queue" err="remote smtp server does not implement smtputf8 extension, required by message, transient, transient" pkg=queue msgid=169 recipient=laurent@example.com backoff=7m28s nextattempt=2024-03-24T17:45:05+01:00 cid=18e66f38ee7 from=robot@infra.example.com attempts=0 cid=18e66f38eec permanent=false code=0 secode=

The email comes from a Prometheus alert.

Prometheus Alertmanager → mox → recipient SMTP server

mox requires the SMTPUTF8 extension on the recipient SMTP server although none of the email headers contain non-ascii characters. I've checked files in the data/queue/a/ folder, none of those files contains non-ascii characters. The command grep -r -P "[\x80-\xFF]" data/queue/ gives no results.

Prometheus uses the golang SMTP Client. This SMTP client uses the SMTPUTF8 extension when the SMTP server supports it (which is the case with mox) even when it's not needed (see: https://pkg.go.dev/net/smtp#Client.Mail).

It seems that mox logs somewhere that the email sent by Prometheus needs the SMTPUTF8 extension, and then requires that the recipient SMTP server also supports this extension. Not sure about that, is this assumption correct?

Could it be possible that mox actively checks if the SMTPUTF8 extension is really mandatory (not rely on the received email), and that it does not requires this extension when it's not needed?

[mjl-]

Good one, thanks for reporting! I don't know what the required behaviour is. But the helpful behaviour is indeed to see that there is no utf8 in the headers (or smtp mail from/rcpt to command). Indeed we should check if that's the case, and not require smtputf8 when delivering. There could be some trickiness involved for multiparts, which can have their own header, possibly with utf8 (e.g. file names of attachments). But perhaps those aren't part of the headers smtputf8 is about.

If you have time & interest to try to implement this, I can give pointers. Otherwise I can probably get to it in the next few weeks.

[lmeunier]

I've some spare time to implement this. Pointers will be appreciated :)

[mjl-]

great!

smtpserver/server.go is where incoming messages are accepted (submission), and added to the queue for delivery. our current behaviour: when we see the client (submission) using the smtputf8 extension, we keep that flag and pass it on to the queue where it will be delivered with that flag.

type "conn" is a connection. we keep track of the smtputf8-enabled flag: https://github.com/mjl-/mox/blob/40ade995a5e5cc40bc8cf8fb3b94e4e508079097/smtpserver/server.go#L334

that line already mentions we should be doing the smtputf8 decisions ourselves.

we are copying the field when the extension is used:

https://github.com/mjl-/mox/blob/40ade995a5e5cc40bc8cf8fb3b94e4e508079097/smtpserver/server.go#L1421

the message is added to the queue with the connection's smtputf8 flag in the submit() function (called from where the smtp DATA command is handled):

https://github.com/mjl-/mox/blob/40ade995a5e5cc40bc8cf8fb3b94e4e508079097/smtpserver/server.go#L2013

the fix will be to get that smtputf8 flag to false when there is no utf8 in the headers, and not in the localparts of the smtp "mail from" and "rcpt to" addresses (the domain could have utf-8, because we can always encoded that as punycode). those addresses are in "type conn" too, they're also used in the queue.Msg.

it may make sense to modify the message.From function to return the parsed message.Part. it already parses the message. message.Part has a method HeaderReader() from which you can read through the header looking for non-ascii bytes.

https://github.com/mjl-/mox/blob/40ade995a5e5cc40bc8cf8fb3b94e4e508079097/smtpserver/server.go#L1885

https://github.com/mjl-/mox/blob/40ade995a5e5cc40bc8cf8fb3b94e4e508079097/message/from.go#L28

further notes: in the submit code path, the field c.smtputf8 is used a few times to compose headers: if smtputf8 was used, it can cause internationalized domain names to be in utf-8 instead of ascii/punycode. ideally, we would decide as early as possible whether we need smptuf8 for the outgoing message, and then pass that decision to the functions doing the formatting. then we still get utf8 headers if we need to any, but we stay ascii-only if we don't need to.

hope this can get you started with the code. i can give more hints on development practices, though just "make build test" should get you started. using "mox localserve" for local development can be nice: easy to setup, and at least the behaviour up to adding messages to the queue will be as a regular mox instance (mox localserve does not deliver messages from the queue, it just delivers to the account that added the message to the queue).

[mjl-]

And another thought: We could also just try to deliver a message that (thinks it needs) smtputf8 if the remote server doesn't claim to support it. As long as smtputf8 isn't needed for a localpart in the smtp MAIL FROM and/or RCPT TO commands, there's a good chance some mail servers won't even care and just pass along the data. From the error message, it seems we're now giving up before trying.

[lmeunier]

Thanks for all these informations.

FYI, I had to run make jsinstall before building mox with make build test. After that, mox was successfully build, tests are all ok, mox localserve is working as expected and I was able to use the local mox instance to send an email with the mox@localhost account on port 1587.

Now I'm ready to start fixing this issue :)

Question: how do you debug a running mox process? I used go build -gcflags="all=-N -l" && ./mox localhost in a terminal, and then connect to the running process with vscode, but there is maybe a better option.

[mjl-]

I pretty much never debug running mox processes. I'm usually debug based on logging. I can imagine an IDE like vscode has a button to "build, run and debug".

I'm updating the develop.txt file with instructions for a full build and running tests, including installing the js dependencies.