search

fyiorgnz / alaveteli

OIA/LGOIMA (Freedom of Information) request system: New Zealand fork of mysociety/alaveteli
https://fyi.org.nz/
Other
4 stars 3 forks source link

Migrate from Sendgrid to SES #1

Open nigeljonez opened 5 years ago

nigeljonez commented 5 years ago

Sendgrid is becoming problematic due to reputation issues (Spamhaus) and resulting delivery reliability.

Plan: Migrate from Sendgrid to SES

Blocking issues:

nigeljonez commented 5 years ago

Although the backscatter hasn't caused us problems, implementing SES means we should also tackle it by avoiding Alaveteli softbounces, our sources appear to be:

Source Current Proposed
Invalid To: addresses that don't match a request Route to holding_pen pgsql Lookup view to validate prefix-id-idhash combinations and hard-reject if invalid
Requests marked authority_only Default is to software bounce Code change to set holding_pen as default, and DB migration to retire bounce
Old requests Migrated to authority_only (old months old), Migrated to nobody (very_old months old) (src) No action required, defaulting to holding_pen will avoid bounces
nigeljonez commented 5 years ago

The issue with e-mails to Ministers may be related, in case it is, this is now high priority.

olineham commented 5 years ago

Backscatter problem has been temporarily solved by updating all requests from bounce to holding_pen. Sendgrid stats show the huge surge of blocks/bounces and overall volume has been eliminated. This is at the expense of a big increase in the number of holding pen emails to deal with. I've opened #8 to discuss longer term options.

olineham commented 5 years ago

Inability to tag certain e-mails with SES ConfigurationSets: is this truly a blocker for SES migration? A single configuration and no tagging is already the situation with Sendgrid. Certainly good to have for SNS but we'd be no worse off than we were before without it.

Logging, on the other hand, is an absolute requirement. We need to keep evidence (even if found by manual search) whether a given outgoing email was delivered.

Postfix config accepting invalid hashes: also not a blocker I think since backscatter is gone (in the short term). I've opened #9 for this as an independent enhancement.