search

pentacent / keila

Open Source Newsletter Tool.
https://keila.io
GNU Affero General Public License v3.0
1.38k stars 70 forks source link

Bounces handling for SMTP senders #316

Closed almereyda closed 3 months ago

almereyda commented 3 months ago

https://github.com/pentacent/keila/issues/221#issuecomment-1637688079 reports that bounces handling is only implemented for Amazon SES senders. For those who use the other mailing APIs or their own SMTP servers, there is no bounce handling implemented right now.

In the database tables of our email server, we can see that a number of deliveries bounced, yet this information is not fed back to Keila on a Return-Path and there is no logic to accommodate for presence of bounce messages in IMAP or POP3 inboxes.

wmnnd commented 3 months ago

What you can already do is handle bounces externally and use the Keila API to adjust the contact status accordingly.

In order to achieve bounce support for SMTP senders, we’d have to add an SMTP server to Keila or add an IMAP/POP3 client to read bounces from an external inbox. That would be quite a lot of work for which we don’t have the resources at the moment. If this is a feature you need for your own project with Keila, please feel free to reach out privately via email.

Adding bounce support for other API providers would be much easier and PRs are welcome :blush:

almereyda commented 3 months ago

Thanks for letting me know.

Our email server actually has an API and the database contains entries for bounced messages sent from the sender account. It is possible to parametrise the search endpoint for the message log in a way to poll bounces for the sender.

Can Keila fire a webhook when sending is done, in so a workflow can be triggered, which pulls the data and updates the status in the Keila API?

When sketching the API surface of SES that Keila uses, very much like Minio mimics the S3 API, how would a similar transactional email service be integrated into Keila? Which endpoints are involved locally and on the remote? I have never used SES and don't know what these SNS notification subscriptions are.

Is this some kind of message bus to which Keila subscribes and from where it receives bounce notifications? Maybe Gotify could help us out here in self-hosted scenarios, given the transactional SMTP email provider knows how to produce such notifications and push them to the bus.

An alternative could be to build something around another common FLOSS transactional mail API like Postal. It knows to fire Webhooks, when a bounce is encountered. Webhooks - Postal - the open source mail delivery platform

We'd only need to learn how to interpret and transform it to feed it back to Keila. It appears possible that this can be done with little glue plugging with an external workflow automation service, such as n8n (not FLOSS anymore), activepieces, windmill and the likes.

wmnnd commented 3 months ago

Keila doesn’t actively subscribe to an API but it can consume SES webhooks.

The workflow you described where Keila triggers a webhook after sending doesn't work, btw because bounces are asynchronous and it can take several hours after sending before a bounce has arrived.

So one option would be to offer a generic webhook in Keila to consume bounces.

almereyda commented 3 months ago

Should I write a new story or we reopen here and rename this one?

dompie commented 3 months ago

Another solution that might be easy to implement:

The workflow in my usecase is:

  1. send newsletter
  2. IT dept collects bounces
  3. IT dept puts bounces in CSV File once a day
  4. I put them in a bounce table
  5. disable keila contacts depending on bounce table contents ( we have a "keep until" column)
wmnnd commented 3 months ago

Is there a standardized file format for bounces?

almereyda commented 3 months ago

There are some kind of control codes, which can be included in "non-delivery notifications".

RFC 5321: Simple Mail Transfer Protocol sends me to RFC 3461: Simple Mail Transfer Protocol (SMTP) Service Extension for Delivery Status Notifications (DSNs) and RFC 3464: An Extensible Message Format for Delivery Status Notifications. Esp. the latter has more mentions of "non-delivery" in the text.

Bounce Message – Wikipedia (de) also sends me to RFC 3463 - Enhanced Mail System Status Codes, which can maybe help detect bounces already while sending.

Bounce message - Wikipedia (en) ultimately sends me to RFC 6522 - The Multipart/Report Media Type for the Reporting of Mail System Administrative Messages and RFC 5337 - Internationalized Delivery Status and Disposition Notifications.

The GitLab source code also speaks of RFC 3834: Recommendations for Automatic Responses to Electronic Mail and the Auto-Submitted header.

I sometimes like to look at other implementations, to get some inspiration.

Sight seeing around the block.

- [bounce -debounce -pgbouncer · Search · GitLab](https://gitlab.com/search?group_id=9970&nav_source=navbar&project_id=278964®ex=true&repository_ref=master&search=bounce+-debounce+-pgbouncer&search_code=true) - [Setup GitLab MailGun endpoint for syncing bounced invite emails - redo2 (!68307) · Merge requests · GitLab.org / GitLab · GitLab](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68307/diffs) - [lib/gitlab/email/receiver.rb · 2b5baf1c96ea4268ddd2785fe57c50ffe63558a4 · GitLab.org / GitLab · GitLab](https://gitlab.com/gitlab-org/gitlab/-/blob/2b5baf1c96ea4268ddd2785fe57c50ffe63558a4/lib/gitlab/email/receiver.rb#L184-197) - [src/mailman/app/bounces.py · master · GNU Mailman / Mailman Core · GitLab](https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/app/bounces.py) - [src/mailman/app/tests/test_bounces.py · master · GNU Mailman / Mailman Core · GitLab](https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/app/tests/test_bounces.py) - [src/mailman/app/docs/bounces.rst · master · GNU Mailman / Mailman Core · GitLab](https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/app/docs/bounces.rst) - [src/mailman/runners/bounce.py · master · GNU Mailman / Mailman Core · GitLab](https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/runners/bounce.py) - [src/mailman/runners/tests/test_bounce.py · master · GNU Mailman / Mailman Core · GitLab](https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/runners/tests/test_bounce.py) - [src/mailman/interfaces/bounce.py · master · GNU Mailman / Mailman Core · GitLab](https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/interfaces/bounce.py) - [src/mailman/model/bounce.py · master · GNU Mailman / Mailman Core · GitLab](https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/model/bounce.py) - [src/mailman/model/tests/test_bounce.py · master · GNU Mailman / Mailman Core · GitLab](https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/model/tests/test_bounce.py) - [src/mailman/model/docs/bounce.rst · master · GNU Mailman / Mailman Core · GitLab](https://gitlab.com/mailman/mailman/-/blob/master/src/mailman/model/docs/bounce.rst) - [Code search results](https://github.com/search?q=repo%3Adiscourse%2Fdiscourse+bounce_score&type=code&p=1) discourse/discourse `"bounce_score"` - [listmonk/docs/docs/content/bounces.md at master · knadh/listmonk](https://github.com/knadh/listmonk/blob/master/docs/docs/content/bounces.md) - [listmonk/internal/bounce/bounce.go at master · knadh/listmonk](https://github.com/knadh/listmonk/blob/master/internal/bounce/bounce.go) - [listmonk/cmd/bounce.go at master · knadh/listmonk](https://github.com/knadh/listmonk/blob/master/cmd/bounce.go) - [listmonk/internal/core/bounces.go at master · knadh/listmonk](https://github.com/knadh/listmonk/blob/master/internal/core/bounces.go)