Call webhook when email bounces

[cuu508]

Hello, I'm looking into using mox for sending transactional email from a SaaS web application. For handling transactional email, I would need some way to get notified when an email bounces, so the web application can take appropriate action (suppress sending to the bounced address for some time period or indefinitely, show email delivery status in UI, use a fallback email address, etc.). In transactional email services this is typically handled via webhooks: when an email bounces, the service calls a configured webhook with bounce details in the HTTP POST request body. The webhook address can either be pre-configured, or come from a designated email header.

@mjl- would you be interested in having something like this in mox? And, more generally, is the SaaS transactional emails use case something you would be interested in targeting? Thanks!

[mjl-]

hi @cuu508! i think this would be useful to have in mox at some point in the future. the "not supported but perhaps in the future"-list in the README has this: "HTTP-based API for sending messages and receiving delivery feedback". i.e., it's currently quite low on my priority list.

it wouldn't hurt to start thinking about how this feature would work. we should probably look at how existing cloud email providers are doing this. perhaps there is even a defacto standard for webhook URLs and bodies? some questions that pop up:

• when including a special email header for a webhook callback, i suppose it must be removed by mox from the outgoing message before delivering. with a http-based API to submit email, the URL could be added in that http call. with smtp/submission, an account could have a webhook URL that is called for all messages, with a Message-ID (that you add to your outgoing messages) to match on.
• would you expect webhook calls for delays? e.g. if the outgoing queue tries to deliver and encounters a "temporary error", the queue will retry again later. it's quite common that this happens for the first try due to greylisting. mox currently generates a "delivery delayed" DSN after a few temporary failures, and could make a webhook call at that moment too, or possibly for every temporarily failed delivery attempt.
• i suppose you would want to get a webhook call when the remote system has accepted an email. but, due to relaying that could happen by the receiving system, actual delivery to the mailbox may fail later on, and we may receive a DSN somewhat later that about the failure. that would result in another webhook call for the same message that delivery failed.
• what kind of fields are common in webhook callbacks? i presume status, addresses to which delivery failed, including permanent/temporary status, and full smtp status codes.
• is any special action expected to e.g. replies to transaction emails? would a mox account/address be used for regular sending/receiving of email? i'm not a fan of noreply-addresses (that don't exist and will hard-fail delivery). mox verifies if an account is allowed to send with the "message from" address, so it's currently not possible to use a noreply address that doesn't exist.

mox wold need a few new configuration options per account, and probably store more information with messages in the outgoing queue, and possibly for a longer period (also after delivery), e.g. to prevent duplicate webhook callbacks, and potential abuse.

[cuu508]

Ah, sorry, I should have read README more carefully :-)

I'm not aware of a standard for delivery and bounce notifications. Each service seems to use their own status classification, and their own mail header names. For the services I've looked at, the common features seem to be:

• status notifications are delivered via webhooks. The webhook URLs are pre-configured in account settings (not specified in email headers)
• there is some way to specify custom data in an email header (or an API field if sending via HTTP API), that later gets passed on in the status notifications

when including a special email header for a webhook callback, i suppose it must be removed by mox from the outgoing message before delivering.

From what I remember, some services strip the special header, and others don't. My personal preference would be:

• The email sending clients should assume the header values could become public. They should use these values for identification, not authentication. They should sign them to guard against tampering.
• The value of the special header is not useful to the recipient (except for debugging during development), so the MTA should strip it out.

would you expect webhook calls for delays?

Yes. The webhook handler can easily ignore the status values it is not interested in. So, the more the merrier.

Some services let you configure what types of events to send. For example, see the screenshot in Brevo docs here. Others send all possible event types, and the webhook handler can sort it out.

i suppose you would want to get a webhook call when the remote system has accepted an email. but, due to relaying that could happen by the receiving system, actual delivery to the mailbox may fail later on, and we may receive a DSN somewhat later that about the failure.

That makes sense. I would write the webhook handler with the assumption that each sent message could result in multiple delivery notifications. It would be good to document the status values. The documentation than then explain that "sent" means "the remote server accepted the message", not necessarily "the message landed in recipient's inbox".

what kind of fields are common in webhook callbacks?

Here are a few specific examples:

Brevo:

• event type (sent, soft bounced, hard bounced, complaint, error, etc.)
• recipient email
• webhook id
• date
• message id
• subject
• X-Mailin-custom (their custom header)
• sending ip
• template_id
• tags

Elastic Email:

• from address
• from name
• envelope from address
• envelope to addresses
• headers
• subject
• body text
• body html
• attachment1 (name and base64-encoded content)
• attachment2 (name and base64-encoded content)
• ...

Sendgrid:

• event type
• recipient email
• timestamp
• message id
• for bounces, the boune type
• reason
• status

Postmark:

• notification id
• bounce type
• recipient email
• from email
• metadata (custom metadata that was included in the email)
• inactive (lets you know if this bounce caused the email address to be deactivated)
• can activate (lets you know if this address can be activated again)

is any special action expected to e.g. replies to transaction emails?

In my case, I use Fastmail for receiving email, and for writing customer support emails. mox and Fastmail would send from the same domain, and both would be listed in the SPF record. When somebody hits "Reply" on an email sent by mox, I would receive in the Fastmail inbox.

[mjl-]

hi @cuu508, thanks for the research and pointers, and sorry for the delay. i did read your response back then. webhooks is now higher on the priority list, but probably it will still take a few months before i'll to it.

i may also add a basic HTTP-based API for sending messages. plenty of web apps use that instead of composing email messages themselves and sending them over HTTP. it's not necessarily related to webhooks, but in the area of work, so i wanted to mention it. if you have any ideas on that, i'm interested in hearing them.

[cuu508]

Thanks for the update! I'm currently using maddy, but am open to (re-)evaluating other options.

i may also add a basic HTTP-based API for sending messages. plenty of web apps use that instead of composing email messages themselves and sending them over HTTP. it's not necessarily related to webhooks, but in the area of work, so i wanted to mention it. if you have any ideas on that, i'm interested in hearing them.

I'm personally happy with using SMTP, but I'm sure in certain scenarios (say, serverless functions, or quick shell scripts) an HTTP API would be handy. For the quick-shell-scripts case, it would be neat if the API invocations were as minimalistic as possible. As an example, I think Mailgun's HTTP API is reasonably simple:

https://documentation.mailgun.com/en/latest/quickstart-sending.html#send-with-smtp-or-api

[mjl-]

@cuu508 it's been almost a year since you created this issue, but i've finally gotten around to it.

the commit above (with tests fixed in subsequent commit) brings a basic http/json-based webapi (to send messages, and make some changes to stored incoming messages, like setting/clearing message flags/moving to mailbox/removing message), and webhooks for outgoing deliveries and for incoming deliveries.

the webapi docs start at https://pkg.go.dev/github.com/mjl-/mox@v0.0.11-0.20240416121811-daa88480cb69/webapi.

i looked at the api's you mentioned, and merged it into an approach for mox to use. will be interested in feedback about the new functionality, api, and docs about it.

i wrote https://github.com/mjl-/gopherwatch, running at https://www.gopherwatch.org, to help me understand what a developer needs from an api like this. has been helpful as guidance too. the code still has the compose/smtp/imap approach i started with, along with the mox webapi/webhook approach.

[cuu508]

@mjl- awesome to see this. I'll try to find time to try this new functionality out. I'm not looking to migrate away from maddy in the near term, but it would be only fair to test the functionality that I myself proposed :-)

From the initial look at the documentation, all seems well explained. Two small observations:

• " Webhook delivery failures are retried at a schedule similar to message deliveries, until permanent failure. " – from this alone it was not clear to me when a permanent failure is declared. Is it after x failed deliveries (what is x?), or is it after the webhook target returns a specific status code like 404?
• "Webhooks for outgoing delivery events and incoming deliveries are configured per account." – from this alone it was not clear where webhooks are configured – in a configuration file, or through an UI. But I assume this would become clear as soon as I start to set up mox, look at configuration examples, and poke around the UI.

[mjl-]

@cuu508 great, thanks. all feedback is appreciated. (:

i'll update the docs soon to address your findings. the intervals are: immediate 1m 2m 4m 7.5m 15m 30m 1h 2h 4h 8h 16h. the config is in the domains.conf (dynamic config), which can be changed by editing the file or through the account web interface (which just writes out a new domains.conf). there is no special handling of http status codes yet. i plan to immediate permanently fail webhook delivery for 403 errors.

it should be easy to use "mox localserve" locally, the webhook (and queue) functionality works with localserve too.

[cuu508]

I tried out the webhook functionality today.

For context, I briefly tested mox last year. I've forgotten most of that, so I'm going in almost fresh. I am familiar with email basics. My objective is to evaluate if mox would work for delivering transactional emails from a SaaS web app.

• I created a new VPS running Ubuntu 22.04, added a DNS "A" record for it
• I installed go 1.22 on it, cloned the mox repo, ran go build in it
• I ran mox quickstart, which gave me a list of DNS records to add, and credentials for accessing admin UI
• I ran mox serve, set up a SSH tunnel to the VPS, and logged into the admin UI
• It became obvious where an how to set the webhooks up :-)
• I added most of the DNS records
• I configured a local instance of my web app to use mox as the SMTP gateway
• I sent a test email from my web app to a gmail.com address through mox. The email arrived, and a "delivered" webhook fired

So in summary all went pretty well so far. As a next step, I would need a way to associate an email sent from the web app with a webhook from mox. i.e., I need some way to pass some token when sending an email, and receive it back in the webhook payload. I see a FromID field in the webhook payload, but for me it comes back empty. Can the FromID field be supplied by the SMTP client, if yes, how?

[mjl-]

Nice, thanks for going through this and providing the feedback! Very helpful.

Can the FromID field be supplied by the SMTP client, if yes, how?

Not currently, but indeed this should be possible! So far, I worked with two modes of operation: 1. Use webapi+webhooks. 2. Use smtp/submission+imap (for processing DSN messages for delivery feedback). You're working in what is probably the most common mode: 3. smtp/submission+webhooks.

You can get FromID filled by setting the "FromIDLoginAddresses" config field for an account in domains.conf. It's the field "Unique SMTP MAIL FROM login addresses" in the account web page. If that's enabled, it causes mox to generate a random FromID for each recipient during submission. And that FromID will be sent back in the webhook call. But the problem with SMTP is that it has no way to provide feedback about a submitted message (need to specify extension for that!). It makes sense to allow you to generate your own FromID and use it in the SMTP MAIL FROM, and see it back in the webhooks. Mox generates a FromID per recipient, because we want to match DSNs back to an original recipient, so for submission with your FromID, we would probably require you only submit to one recipient in the transaction (which I think is the normal situation in transactional mail). And you would be responsible for ensuring the value is indeed unique.

Summarizing: In submission, you would login with an address configured in FromIDLoginAddresses, use a MAIL FROM:<you+<fromidyougenerated>@domain.example>, have only a single allowed recipient in that transaction, and you'll see webhook calls with FromID set (and QueueMsgID will be set whenever FromID is set).

Your domain needs to have a "localpart catchall separator" configured (e.g. "+") for FromIDs to work. For matching incoming DSNs, you need to configure "Keep messages retired from queue" in the account page (config option "KeepRetiredMessagePeriod").

Some more info about FromID and matching DSNs back to their original transactions:

If you have an account with one address, and all outgoing mail should be using unique from addresses, you can just specify that single address in the "unique smtp mail from login addresses". In mox, you can authenticate with any address you own, including variants using the catchall separator. So you could also configure "you+fromid@domain.example" and authenticate with that login name during submission of transactional mail. Then regular email from a mail client/webmail will still be sent with a regular non-unique SMTP MAIL FROM.

I've struggled a bit with this behaviour, and how to configure it, and not confuse users. I'm not too happy with it. I should probably at least add the word "FromID" to the "Unique SMTP MAIL FROM login addresses" section on the webaccount page. Suggestions welcome on how to make this more clear.

To match webhooks to your messages, you could also add your own extra data by adding "X-Mox-Extra-: " headers to submitted messages (those headers aren't currently deleted from the message). Those will be present in the webhooks for delivery events generated directly by mox (e.g. delivered/delayed/failed to next-hop MX host) if FromIDs are enabled for the transaction. But for DSNs from remote servers (for messages that have left our queue), the FromID is currently needed to match incoming DSNs back to the original outgoing transaction. In the future, mox should use the SMTP DNS extension with the ENVID parameters (it will work in many cases, but it's an optional extension, not everyone implements it; mox doesn't at the moment!). Likewise, mox should try to match incoming DSNs based on the original message-id (which is optional in DSNs, and doesn't uniquely identify a recipient in case of multiple recipients in a transaction). So, there are other options for matching DSNs to original outgoing messages, they may be easier to use than "FromIDs", but probably work a little less reliably.

If you enable FromIDs for your submission, and you add X-Mox-Extra-* headers, you should already get those extra values back in webhooks for delivery events from the queue (but not from DSNs).

[mjl-]

Perhaps it makes sense to add method to the webapi to submit a precomposed message, and that returns mox-generated FromIDs. Would be a relatively easy switch from using submission if a developer wants to do that. The current webapi.Send needs the fields/data that make up a message, but if you're using submission, your compose and submission code is likely separate.

[cuu508]

Summarizing: In submission, you would login with an address configured in FromIDLoginAddresses, use a MAIL FROM:you+<fromidyougenerated>@domain.example, have only a single allowed recipient in that transaction, and you'll see webhook calls with FromID set (and QueueMsgID will be set whenever FromID is set).

Ah, great, thanks! Putting the token in MAIL FROM is what I've already been doing in my current setup.

I added the sender's address (without the local catchall part) in "Unique SMTP MAIL FROM login addresses", and FromID in the webhooks is now populated (with whatever I put in the catchall part).

It's great to also have the X-Mox-Extra- headers as an alternative.