Make email digest unsubscribes permanent

seanh commented 1 year ago

Context

The email digests feature uses Mailchimp Transactional (a.k.a Mandrill)'s automatic unsubscribe footer feature to have Mailchimp automatically add unsubscribe links to the bottoms of the emails for us. Specifically I've added a rule using their rules engine to only add the unsubscribe links to emails that use the instructor_email_digest template, because we don't necessarily want to add the footer to all emails that we send.

Problem

Unsubscribes created by clicking on these automatically-generated unsubscribe links expire after a year by default! This is mentioned in the docs here:

The length of a rejection depends on the type: soft bounces expire after 24 hours, hard bounces after seven days, unsubscribes and spam complaints after a year, and addresses manually added to the Rejection Denylist never expire. After a rejection expires, if an email is sent to the address, Mailchimp Transactional will attempt to deliver the message—but subsequent bounces, spam complaints, or unsubscribes will result in the address being added to the rejection list for a longer period of time.

I've also confirmed with Mailchimp support that unsubscribes do indeed expire after a year.

We need to do something to make these unsubscribes permanent.

Possible Solutions

Here's a few ideas:

If we change to using an unsubscribe merge tag instead of the rules engine to add the unsubscribe footer links then we can have Mailchimp redirect the user's browser to a URL of our choosing after unsubscribing. The way I think this works is that the unsubscribe gets added to Mailchimp (but will expire after a year) and then Mailchimp redirects the browser to our URL. On receiving the redirect we could either:
1. Record the unsubscribe in our own DB as well. Then we'd have to change our email sending code to not send emails to unsubscribed users.
  
  I don't think this would cause synchronization issues between the unsubscribed emails in our DB and the ones in Mailchimp because the two lists don't need to be kept in sync. In order to be unsubscribed an email needs to be in the denylist in our own DB--just being in Mailchimp isn't sufficient as those expire. In order to be subscribed an email needs to not be in either the denylist in our DB or the one in Mailchimp, so if we ever implement a re-subscribe feature it'll have to call the necessary Mailchimp API to make sure the email is not on Mailchimp's denylist. I believe this would be the /rejects/delete API, note that the docs for this API say that each deletion has an affect on your reputation! That doesn't sound good.
2. Alternatively, instead of recording the unsubscribe in our own DB we could call Mailchimp's /rejects/add API to manually add the address to the denylist. Manually-added addresses never expire. I confirmed with Mailchimp support that this approach is possible to turn temporary denials into permanent ones.
  
  I'm not sure what happens if you try to manually add an address to the denylist when that address has already been (temporarily) automatically added to it. Will we need to remove the address first then add it?
  
  Is there a race condition if we manually add the address to the denylist permanently and then Mailchimp automatically adds it temporarily, turning our permanent denial into a temporary one?
Other problems with this approach:
1. From the Mailchimp docs I don't see how these redirects are authenticated.
Instead of redirects we can use Mailchimp's webhooks to get notified when users unsubscribe. This way we could continue to use the rules engine to have Mailchimp add the unsubscribe links, we wouldn't have to change to using merge tags instead. Unlike with their redirects, Mailchimp's docs do explain how authentication works with webhooks. On receiving the webhook we'd either record the unsubscribe in our DB or call the Mailchimp API to record a permanent unsubscribe in Mailchimp, same as with redirects above. If recording the permanent unsubscribe in Mailchimp I have the same concern as above about a potential race condition.

Other problems with this approach:
1. We need to create the webhook. This can be done manually in their UI or it can be done by calling the add webhook API. I don't think we'd want this to be a manual step when deploying the LMS app--if a new instance of the LMS app is ever deployed one day someone will surely forget to create this webhook and not notice. But if we want the app to create its own webhook automatically when is it going to do that? Every time the app starts up (which would be once per EC2 instance per deploy, plus autoscaling etc)? Have a periodic task for doing it? Either way will the code have to avoid creating multiple webhooks by checking whether a webhook already exists and only creating one if it doesn't? If so then do we have parallelism issues e.g. if multiple app instances are trying to upsert a webhook at the same time?
Implement a periodic Celery task (via h-periodic) that downloads the rejection denylist from the Mailchimp API and either makes further API calls to make any automatic temporary denials into permanent ones or records those denials permanently in our own DB.

The main question with this approach seems to be whether we can actually get the entire rejection denylist our of the Mailchimp API? There doesn't appear to be a simple paginated API for stepping through the entire denylist. There is a /rejects/list API but it only returns up to 1000 results (and which 1000? the docs don't say and there's no sort parameters). There's also a /exports/rejects API, maybe that would work.

Note that we'd want to limit it to digest email unsubscribes only. The /rejects/list API has a subaccount parameter that it looks like we could use this for. I'm not sure about the /exports/list API.
We could implement our own unsubscribe links feature and not using Mailchimp's unsubscribe links at all.

This means we'd have to design and implement a feature for generating email unsubscribe links. I'm not sure what the best practices here are. Single-use and/or expiring links? The links certainly need to be unguessable, we don't want an attacker to be able to go through unsubscribing everyone. Each time we send an email we'd need to generate the unsubscribe link and include it in the email (in the template variables that we send to Mailchimp). When receiving an unsubscribe request we'd need to record these in our DB. Our email-sending code would need to respect these unsubscribes. Finally we'd need to render a "You have been unsubscribed" page in response to each unsubscribe request.

Aside from the potentially large amount of work a downside of this approach is that we'd lose features that the Mandrill dashboard provides around tracking our unsubscribe rate etc.

marcospri commented 1 year ago

Webhooks are awkward to develop with unless lots of attention is paid to by the service (thinking about Stripe here, they take care of that). There's also the extra complexity of new environments. Not the end of the world but having a manual step there it's not ideal.
Retrieving the full list seems to be unnecessarily hard. Either deal with the endpoint returning only 1000 entries or the export one scheduling an export of a zip file. Neither sounds ideal.
The redirect seems doable but I reckon the amount and nature of work would be similar to rolling our own solution from scratch.

That leaves writing our one unsubscribe button, did a first PoC of that here: https://github.com/hypothesis/lms/pull/5259

Not a lot of code although it's missing some error handling etc.

We'd miss the UI from Mailchimp around unsubscribes, we'd have to do with some metabase queries.

We'll still get open/click rates from Mailchip and handling of bounces/spam from then. In return, we get a more provider-agnostic implementation of this although we might never change providers.

marcospri commented 1 year ago

Done in https://github.com/hypothesis/lms/pull/5275

hypothesis / lms