magicdude4eva / port25-bouncehandler

Port25 PowerMTA bounce handler for Interspire and MailWizz
MIT License
73 stars 42 forks source link
bounce-handling fbl interspire mailwizz mx-record port25 powermta

Port25 PowerMTA bounce handler for Interspire and MailWizz

paypal


:beer: Please support me: Although all my software is free, it is always appreciated if you can support my efforts on Github with a contribution via Paypal - this allows me to write cool projects like this in my personal time and hopefully help you or your business.


TL;DR - Features

IMPORTANT: Before logging issues

I receive frequent requests to provide consulting- or installation-services for Port25, the scripts etc. While I appreciate your donations to keep my contributions going, I can unfortunately not assist you with fixing your configuration / installation.

The majority of issues I receive are based on:

IMPORTANT2: Don't resell my scripts

The scripts are open-source and free. There is nothing more annoying when I receive emails / issues from your angry customers that they paid ridiculous amounts of money for bounce-handling and my scripts are not working. If you resell my scripts and then change license information / remove any attribution to my work you are violating the license of this project.

If you resell my scripts, keep copyright intact and then perhaps also extend the decency to provide a Paypal contribution. I am more than happy to fix bugs and make improvements, but will not do so on the back of your commercial ventures.

Overview

Transactional- and promotional mail to our customers is an important mechanism to stay in touch and hopefully improve revenue due to the value and information we provide in our communication with customers.

But let's be honest: Bounce handling and keeping your mail-lists is tough. Free solutions do not seem to exist and professional solutions/services are expensive. I have never really understood this, as bounce-handling is generally quite easy:

Once our MTA (mail transfer agent) such as PowerMTA records a hard-bounce we need to remove the recipient email from our system as it not just wastes resources attempting to to send mail to a non-existant email, it will eventually hurt sender reputation.

The big challenge comes in that campaign tools (especially when hosted in-house) either require "dedicated bounce email accounts" or require software development skills to write code to integrate with the campaign APIs to manage unsubscribes.

This Port25 PowerMTA bounce handler addresses this as it can run standalone (you provide a CSV) or it can integrate with PowerMTA acct-file pipe feature and unsubscribes via API calls to MailWizz, Interspire and any other provider in near-realtime.

:warning: I run OS X and Linux and I have not tested any of it on Windows. I suppose that Windows will play nice with PHP, but if you find any issues, please log it and I will fix with your help.

Installation

The installation is quite simple:

setup.php

All configuration is controlled within setup.php and the following options can be adjusted:

DEFAULT OPTIONS:

INTERSPIRE CONFIGURATION: If you do not use Interspire, leave constants as undefined:

MAILWIZZ CONFIGURATION If you do not use MailWizz, leave constants as undefined:

About Bounce processing

For me a mail-bounce means that we will never, ever be able to deliver to the recipient again for various reasons (mailbox does not exist, domain does not exist etc). There are many cases where free-mail (such as Nokia Mail) cease to exist and in such cases the recipient should be unsubscribed from ALL CONTACT LISTS. Email marketers freak out about such "bold" move, but to be honest if an email bounces in one contact list, it is pointless to hold on to that subscription in other contact lists.

My approach is that if a delivery permanently fails, we unsubscribe the recipient from all systems. On our transactional enviroment (where users continue to log in with their non-functioning email-address/userid) we show the user a notice that we had delivery issues and ask the user to reconfirm his email. This is an elegant and efficient way to reconfirm email details.

About Interspire bounce processing

Interspire allows the management of multiple contact lists which are used for various campaigns. Naturally an email-recipient can be subscribed to one or more contact lists.

When we bounce an email-recipient in Interspire, we subscribe the email-recipient from all contact-lists, irrespective from where the bounce originates from. Interspire does not allow a "bulk-unsubscribe via email-address" and we have to run several API calls to unsubscribe a single recipient.

:warning: We cache the Interspire contact lists upon startup. In a future release I will adjust this to periodically refresh the cache to avoid a scenario where newly created lists are not refreshed.

About MailWizz bounce processing

MailWizz functions similar to Interspire and we also unsubscribe an email-recipient from all contact lists.

In order for the port25-bouncehandler to properly identify MailWizz campaigns, you need to configure a custom-header in MailWizz via "Servers -> Delivery Servers: Additional Headers" as depicted below (this needs to be done per delivery server):

MailWizzJobHeaderForPort25

Standalone processing

You can manage bulk-unsubscribes via standalone. The only pre-requisite is a CSV file which contains an email-address in the first column of the file.

:warning: With Standalone processing we will always unsubscribe from all configured providers, so make sure that your CSV file is correct. We do however log from which lists a recipient is unsubscribed, so in case something goes wrong, you can find out which addresses are affected.

Command-line options

To run the standalone processing you simply pipe the CSV file into the bounce-handler:

cat bounce.csv | /usr/bin/php ./bouncehandler/bouncehandler.php

This will then output progress into the console or log-file:

[29/May/2016:09:46:44] Port25 PowerMTA bounce-handler
[29/May/2016:09:46:44] (C) 2016 Gerd Naschenweng  http://github.com/magicdude4eva
[29/May/2016:09:46:44] ------------------------------------------------------------------
[29/May/2016:09:46:44] Handling bounce categories=bad-mailbox,bad-domain,routing-errors,inactive-mailbox
[29/May/2016:09:46:44] Bounce-provider: Interspire, initialising
[29/May/2016:09:46:44]    Endpoint-URL=http://interspire.example.com/xml.php
[29/May/2016:09:46:45]    Interspire enabled with lists=112,3,32,95,81,108,109,115,116,117
[29/May/2016:09:46:45] Bounce-provider: Interspire, complete
[29/May/2016:09:46:45] Bounce-provider: MailWizz, initialising
[29/May/2016:09:46:45]    Endpoint-URL=http://mailwizz.example.com/api
[29/May/2016:09:46:45]    MailWizz enabled!
[29/May/2016:09:46:45] Bounce-provider: MailWizz, complete
[29/May/2016:09:46:45] Starting bounce processing
[29/May/2016:11:22:09] Starting bounce processing
[29/May/2016:11:22:10]   MailWizz: unsubscribe for XXX@domain.com:
[29/May/2016:11:22:10]    - skipped: A Mailwizz list #1
[29/May/2016:11:22:10]    - skipped: A Mailwizz list #2
[29/May/2016:11:22:11]   Interspire: Skipping recipient XXX@domain.com - no subscribed lists returned
....
[29/May/2016:11:22:12] Completed bounce processing! Total records=4, processed=4, skipped=0

The Standalone processing can also be used to process PowerMTA files without using the acct-file-pipe-processing. It is perhaps something you should look at before automating it to test that integration works and that your bounce file is correct.

Running a PowerMTA file in standalone processing is the same command as above:

cat pmta-bounce-file.csv | /usr/bin/php ./bouncehandler/bouncehandler.php --debug

Note that if your setup does not follow the recommendations for the acct-file and your columns are in different sequence, you will have to change the PORT25_OFFSET_*-constants in bouncehandler.php.

The processing of a PowerMTA file is quite similar, with the only addition that the record and bounce-category is also logged:

...
[29/May/2016:11:28:04] Bounce: bad-domain from=mailwizz@campaign.com via vmta-my-mta01/XXX@domain.com
[29/May/2016:11:28:06]   MailWizz: Skipping XXX@domain.com, already unsubscribed!
[29/May/2016:11:28:06] Bounce: bad-mailbox from=mailwizz@campaign.com via vmta-my-mta01/YYY@anotherdomain.com
[29/May/2016:11:28:06]   MailWizz: unsubscribe for YYY@anotherdomain.com:
[29/May/2016:11:28:06]    - skipped: MailWizz list name
[29/May/2016:11:28:06] Bounce: bad-domain from=interspire@campaign.com via vmta-my-mta01/WWW@domain.com
[29/May/2016:11:28:07]   Interspire: Skipping recipient WWW@domain.com - no subscribed lists returned
...

Port25 processing

:warning: I can not help you with configuration/setup issues when it comes to PowerMTA. I have extensive experience, but each setup varies, although the basic principles remain the same. This README is comprehensive enough that you will be able to configure your environment properly if you read this document twice.

If you do not follow the default setup below, I suggest you read the PowerMTA user-guide (specifically: 3.3.7 Accounting Directives and 11. The Accounting and Statistics).

Process manually first

My suggestion is that you first start off with manual processing, by just adding the following section to your /etc/pmta/config file.

<acct-file /var/log/pmta/bounce.csv>
    delete-after 60d
    move-interval 5m
    max-size 500M
    records b
    record-fields b timeQueued,bounceCat,vmta,orig,rcpt,srcMta,dlvSourceIp,jobId,dsnStatus,dsnMta,dsnDiag
</acct-file>

This will generate an accounting file for just bounced records and will look something like this:

type,timeQueued,bounceCat,vmta,orig,rcpt,srcMta,dlvSourceIp,jobId,dsnStatus,dsnMta
b,2016-05-29 01:10:51+0200,bad-domain,vmta-mymta04,mailwizz@example.com,XXX@domain.com,sourcemta.domain.com (0.0.0.0),1.1.1.1,,5.1.2 (bad destination system: no such domain),
b,2016-05-29 01:10:23+0200,bad-mailbox,vmta-mymta03,mailwizz@example.com,YYY@anotherdomain.com,sourcemta.domain.com (0.0.0.0),1.1.1.1,,5.1.1 (bad destination mailbox address),mx1.destindomain.com (2.2.2.2)

Our bounce-processing relies on the sequence of record-fields in the bounce-file and if you change the order, you will need to change the index of those fields in the constants PORT25_OFFSET_* as defined in bouncehandler.php:

PowerMTA field mapped record Description
type bounceRecord[0] type - always b
timeQueued bounceRecord[1] Time message was queued to disk
bounceCat bounceRecord[2] likely category of the bounce (see Section 1.5), following the recipient which it refers
vmta bounceRecord[3] VirtualMTA selected for this message, if any
orig bounceRecord[4] originator (from MAIL FROM:)
rcpt bounceRecord[5] recipient (RCPT TO:) being reported
srcMta bounceRecord[6] source from which the message was received. the MTA name (from the HELO/EHLO command) for messages received through SMTP
dlvSourceIp bounceRecord[7] local IP address PowerMTA used for delivery
jobId bounceRecord[8] job ID for the message, if any
dsnStatus bounceRecord[9] DSN status for the recipient to which it refers
dsnMta bounceRecord[10] DSN remote MTA for the recipient to which it refers
dsnDiag bounceRecord[11] DSN diagnostic string for the recpient to which it refers

You can then run the bounce-processing manually:

cat /var/log/pmta/bounce-2016-05-29-0000.csv | /usr/bin/php /opt/pmta/bouncehandler/bouncehandler.php --debug

Once you are comfortable with this you can then switch into automatic processing.

Automatic processing

Switching to automatic processing is quite simple, you adjust your current record to the following:

<acct-file |/usr/bin/php /opt/pmta/bouncehandler/bouncehandler.php>
    records b
    record-fields b timeQueued,bounceCat,vmta,orig,rcpt,srcMta,dlvSourceIp,jobId,dsnStatus,dsnMta,dsnDiag
</acct-file>

Ensure that the bouncehandler logs the startup-messages in it's log-file. If this does not happen, then PowerMTA is not able to run the PHP due to possible permission errors (ownership of the /opt/pmta/bouncehandler folder or no executable permissions of the bouncehandler.php).

Port25 feedback loop processing

Port25 is capable of processing feedback loop (FBL) reports. In our case we have automated the FBL processing, where Port25 receives the FBL report, then pipes it into our bouncehandler.php which then calls a feedback-loop processor. We automatically remove any reported email from all systems and notify our Postmaster team via email.

Port25 FBL notification

For the setup to work, the following is required:

<acct-file |/usr/bin/php /opt/pmta/bouncehandler/bouncehandler.php --logfile=/var/log/pmta/fbl-processor.log> records feedback-loop map-header-to-field f header_X-HmXmrOriginalRecipient rcpt # hotmail recipient record-fields f *, header_subject, header_BatchId, header_Message-Id, header_List-Unsubscribe, header_List-Id, header_X-Mw-Subscriber-Uid, header_X-Mailer-LID, header_X-Mailer-RecptId

- Adjust the `feedback-loop-processor.php` to according to your requirements
- Register your address `abuse@fbl.example.com` with the various FBL lists [Word To The Wise - ISP Summary Information](http://wiki.wordtothewise.com/ISP_Summary_Information)

## About the Port25 FBL record fields
In addition to the standard FBL fields, we write out the following fields:

* __header_X-HmXmrOriginalRecipient__: Outlook places the original recipient into a customer header, we map this back to `rcpt`
* __header_subject__: We log the subject as it allows to identify the nature of the emil
* __header_BatchId__: Typically the `x-job-id`
* __header_Message-Id__: The unique message-id
* __header_List-Unsubscribe__: The unsubscribe link - this might be useful to just post a HTTP-GET from Port25
* __header_List-Id__: The `List-Id` - most mailers such as MailWizz use the standard header
* __header_X-Mw-Subscriber-Uid__: The MailWizz Subscriber-UID. We use this to unsubscribe users from MailWizz
* __header_X-Mailer-LID__: The list-id provided by Interspire
* __header_X-Mailer-RecptId__: The Subscriber-UID provided by Interspire

Some FBL providers (such as OpenSRS/ReturnPath) will anonymise the recipient-email address and in those cases we need to use the Subscriber-ID/List-ID to unsubscribe the user.

# Port25 List-Unsubscribe handling
The unsubscribe handling currently only supports MailWizz, but can be extended to anything else. In order for MailWizz to work, you will need install and configure the `mailwizz-unsubscribe-extension`.

![Port25 List-Unsubscribe notification](https://raw.githubusercontent.com/magicdude4eva/port25-bouncehandler/master/port25-list-unsubscribe-notice.png)

Additionally, you need to make the following DNS and Port25 changes:

- Create a FBL domain `fbl-unsub.example.com`

- Create a MX record for `fbl-unsub.example.com` which points to your Port25 server - i.e. `fbl-unsub.example.com MX 1 mailserver.example.com`

- Configure the `fbl-unsub.example.com` domain in Port25 to process unsubscribes:

relay-domain fbl-unsub.example.com

type pipe command "/usr/bin/php /opt/pmta/bouncehandler/unsubhandler.php --logfile=/var/log/pmta/unsubhandler.log --from=""$from"" --to=""$to"""

## How it works
The MailWizz extension modifies the `List-Unsubscribe` to includes mailto and HTTP tags to allow providers such as Google, Microsoft, Yahoo to offer their users an automatic unsubscribe without leaving the email client:

List-Unsubscribe: mailto:[SUBSCRIBER_UID].[LIST_UID].[CAMPAIGN_UID]@fbl-unsub.example.com?subject=unsubscribe, http://YOURMAILWIZZDOMAIN.COM/lists/[LIST_UID]/unsubscribe/[CAMPAIGN_UID]/[SUBSCRIBER_UID]/unsubscribe-direct?source=email-client-unsubscribe-button


 When a user clicks on the "Unsubscribe link", the mail-client will send an email to the `mailto` address. Port25 (via the configuration of the MX record) will accept the email, extract the `from` and `to` details and then invokce the `unsubhandler.php` piping the content of the email received into the handler.

 The `unsubhandler.php` then extracts the subscriber-UID and list-UID from the `to` address and then calls the MailWizz API to unsubscribe the user. Once a unsubscribe was successful, an email is sent to you (you will need to adjust `unsubhandler.php` to change this).

 If you do not use MailWizz and generate your own List-Unsubscribe headers, I found the sequence of `mailto` (first) and `http` (second) to be very important. Google outright refuses to process unsubscribe requests if `mailto` is not first.

 ## IMPORTANT: DKIM signing of additional headers
 If you are not using DKIM to sign your emails, you should revisit your email sender practises. DKIM is *absolutely* necessary for Google Postmaster Tools to function and both `Feedback-ID` and `List-Unsubscribe` need to be signed.

 I have placed default options into a wild-card domain so that it applies to all recipients (also note the optimistic TLS for encrypted outbound mail):

<domain *> ...

dkim-sign yes # DKIM signing on messages dkim-algorithm rsa-sha256 # recommended by RFC4871, default rsa-sha1 dkim-body-canon simple # recommended by DKIMCore (default relaxed) dkim-headers Feedback-ID,List-Unsubscribe # additionally sign the Feedback-ID header

use-starttls yes # Specifies whether PowerMTA should use the STARTTLS extension require-starttls no # We use optimistic TLS - i.e. only use it if the recipient supports it

...


## Installation troubleshooting
The configuration described above is running on a vanilla Centos7 installation with the [Webtatic PHP repository](https://webtatic.com/projects/yum-repository/) and our current version is PHP 5.6.24.

From user feedback so far I noticed the following issues:

### Wrong permissions / ownership of /opt/pmta/bouncehandler
The bouncehandler receives piped data via Port25 and as such needs to run as the same user as Port25. Typically PowerMTA runs as user `pmta`, so make sure to set the following permissions:

chown -R pmta:pmta /opt/pmta/bouncehandler


### Wrong path to PHP
On CentOS7, PHP resides in `/usr/bin/php` and the above configuration is based on that. I have seen some customer configurations where PHP resides in `/usr/local/bin/php` and you would have to adjust the Port25 configuration to reflect that.

## RRD configuration
In a Linux environment, RRD is easy to configure and we use it to log unsubscribes and stats - this is pretty much experimental at the moment.

If you do not need it, just remove the following line from `setup.php`:

// RRD Graphs - requires installation of php-rrdtool - if not defined, it will not be enabled define("RRD_FILE", "/var/log/pmta/pmta.rrd");


The installation of RRD varies depending on your distribution. In our case of CentOS 7, it was as simple as:

yum install rrdtool-devel php56w-devel php56w-pear


In some environments you will need to either additionally install `php-pecl-rrd` or `php-rrdtool`

Once done, you have to make the following change in your `/etc/php.ini`

; Used for Port25 RRD reporting extension=rrd.so


## Our mail- / campaign-environment
I have been asked a number of times what infrastructure we are running, and while I can not go into all the specifics, any decent Linux admin will be able to replicate it:
* Everything is FOSS with the exception of PowertMTA
* We run CentOS 7 in a XEN environment (our own "cloud solution")
* We have a dedciated AfriNIC IP range and dedicate IP blocks for system-, transactional- and promotional mails
* We send CAN-SPAM compliant and use TLS for mail
* All in-house servers use a local Postfix installation which relays to Port25. This is great, as Postfix will queue mail in case of maintenance on Port25
* Port25 runs on a dedicated 4-core/12GB virtual server
* We run the port25-bouncehandler in production with a few small customisations (mostly real-time reporting and daily/weekly MTA reports (`pmtastats` is just an awesome tool for this).
* We also use some custom reporting scripts with the RRD data we generate

## Collecting, parsing and analysing Port25 accounting files
We use Logstash and Graylog for log collection and monitoring. Logstash has a versatile CSV filter which allows the parsing and processing of PowerMTA accounting files into Graylog or Kibana - the GIST can be viewed via the GIST: [Port25 Logstash & Graylog integration](https://gist.github.com/magicdude4eva/5001d3b52743062f6fb28e3a92b7fce4)

## Writing your own API / processor
I often get asked "Can you help me integrate your scripts with product XYZ, it has APIs?". Although I enjoy coding, I don't think it is a good idea to write scripts for an environment which I do not use or monitor. The scripts I provide have been used in our production environment for several years and work reliable because we monitor them and tweak them when necessary.

My script setup is however very pluggable and any PHP developer will be able to write a processor by following these steps:

1. Look at the MailWizz provider for inspiration: https://github.com/magicdude4eva/port25-bouncehandler/blob/master/providers/bounce-provider-mailwizz.php
2. Setup config for your provider similar to: https://github.com/magicdude4eva/port25-bouncehandler/blob/master/setup.php#L64
3. Write the API for unsubscribe for your product - https://github.com/magicdude4eva/port25-bouncehandler/blob/master/providers/bounce-provider-mailwizz.php#L80
4. Initialise the provider in the relevant code bases:
- https://github.com/magicdude4eva/port25-bouncehandler/blob/master/unsubhandler.php#L57
- https://github.com/magicdude4eva/port25-bouncehandler/blob/master/bouncehandler.php#L109
5. Add the API calls to unsubscribe in relevant code:
- https://github.com/magicdude4eva/port25-bouncehandler/blob/master/unsubhandler.php#L167
- https://github.com/magicdude4eva/port25-bouncehandler/blob/master/providers/feedback-loop-processor.php#L85
- https://github.com/magicdude4eva/port25-bouncehandler/blob/master/bouncehandler.php#L203
- https://github.com/magicdude4eva/port25-bouncehandler/blob/master/bouncehandler.php#L232
- https://github.com/magicdude4eva/port25-bouncehandler/blob/master/bouncehandler.php#L258

## Donations are always welcome
:beer: **Please support me**: If the above helped you in any way, then [follow me on Twitter](https://twitter.com/gerdnaschenweng) or send me some coins: 

(CRO) cro1w2kvwrzp23aq54n3amwav4yy4a9ahq2kz2wtmj (Memo: 644996249) or 0xb83c3Fe378F5224fAdD7a0f8a7dD33a6C96C422C (Cronos) (USDC) 0xb83c3Fe378F5224fAdD7a0f8a7dD33a6C96C422C (BTC) 3628nqihXvw2RXsKtTR36dN6WvYzaHyr52 (ETH) 0xb83c3Fe378F5224fAdD7a0f8a7dD33a6C96C422C (BAT) 0xb83c3Fe378F5224fAdD7a0f8a7dD33a6C96C422C (LTC) MQxRAfhVU84KDVUqnZ5eV9MGyyaBEcQeDf (Ripple) rKV8HEL3vLc6q9waTiJcewdRdSFyx67QFb (Tag: 1172047832) (XLM) GB67TJFJO3GUA432EJ4JTODHFYSBTM44P4XQCDOFTXJNNPV2UKUJYVBF (Memo ID: 1406379394)



Go to [Curve.com to add your Crypto.com card to ApplePay](https://www.curve.com/join#DWPXKG6E) and signup to [Crypto.com for a staking and free Crypto debit card](https://crypto.com/app/ref6ayzqvp).

Use [Binance Exchange](https://www.binance.com/?ref=13896895) to trade #altcoins. Sign up with [Coinbase](https://www.coinbase.com/join/nasche_x) and **instantly get $10 in BTC**. I also accept old-school **[PayPal](https://paypal.me/GerdNaschenweng)**.

If you have no crypto, follow me at least on [Twitter](https://twitter.com/gerdnaschenweng).