Set up email server or service

[graphiclunarkid]

We need to set up either an SMTP server of our own, or a third-party service, to send out emails to report subscribers. Below is some discussion from an earlier issue, copied for reference:

@dantheta: By the way - do we have a preferred way of sending email to subscribers? Something like mailchimp API or another service like that, or is it cool to use a PHP mailer class and the local postfix install on the webserver? @JimKillock: That's one for @gwire to answer @graphiclunarkid: MailChimp is hosted in the US, and although that doesn't technically contravene ORG's privacy policy, we're not exactly brimming with confidence in the sanctity of the Safe Harbour provisions either ;-) An alternative might be the German company Clever Reach - I hear they have a good API and I happen to have test keys in hand (for another ORG-related project). Otherwise, ORG uses Engaging Networks' e-Activist for its supporter communications, but I understand integration options are limited (perhaps non-existent). @dantheta: That's cool - I was just aiming to find out what you were already using. I'm equally happy with plain ol' SMTP too. I'm not sure what e-activists' facilities are like for sending personalised notifications to individual users. I can write the database part of the results sender and change notifier leaving the mail transport part of it 'til last, if that would be useful. @mkillock: I have some experience with setting up postfix to be a well behaved mass mailer, if that helps. Need to do things like only send one email per 20secs to AOL and such otherwise they block the server IP for 24 hours! Yahoo like DKIM signatures, some like SPF records and so on. If we do this, might be handy to have a separate sub domain for sending, else will need to track down all the IP addresses that use the main ORG domain, including e-activist's IPs.

[dantheta]

I've put the bare minimum sender config into postfix on the API server. This at least changed the origin domain to blocked.org.uk instead of the system assigned hostname.

[graphiclunarkid]

We could try https://uk.mailjet.com/. It's £5/month for 30k messages/month, has both a RESTful and an event-driven API, and would handle #52 for us too. The company is French.

[JimKillock]

I don't know if we need a separate email service or not: maybe @gwire can say what he thinks. We're using our own servers for staff email SMTP, and also the mailman lists.

[graphiclunarkid]

@gwire has said we should be fine to use ORG's existing SMTP server for this. I think we should do so, at least initially, as we don't have much time and regular SMTP from PHP sounds like a solved problem.

If whoever is working on this wants to email me I will share my login credentials so that we can get cracking. Hopefully the server will let us send as (blocked at) even though we're authenticating as me!

[dantheta]

Hiya -

I can get local postfix on the API server to authenticate and relay through the ORG SMTP server, which keeps things nice and straightforward in PHP land.

I'm working on some of the foundation data handling for the email reports at the moment.

Is the preferred sender address "blocked@blocked.org.uk" or is there another option? I'll be putting it in config anyway, so if it gets changed later it's no big deal.

Daniel.

[graphiclunarkid]

Per the discussion in #2 I think we are using (blocked at openrightsgroup dot org).

Forwarding through our local SMTP server to ORG's would make it easier to change provider later if necessary. Good idea!

[dantheta]

Cool. If you can send me the SMTP auth details, I'll get them plumbed in.

[dantheta]

Postfix set up with authenticated outbound relay. Canonical address mapping set up for the chosen email address. All tested from the command line and working fine. There isn't any enhanced mail queueing at the moment (as @mkillock mentions, AOL and Yahoo are very funny about accepting mail).

[mkillock]

I can tell you what I've learnt doing this for work.

1. Return Path and SenderBase have a database of IP address reputations which some of the larger mail handler's use partly to decide how to deal with mail from that IP. Both have an IP lookup to see how you are doing.
2. More or less all the large email providers will block mail if it arrives too quickly from an untrusted source, so rate limiting by domain name is advisable. AOL & Yahoo are the most fussy in this regard.
3. All the big mail providers have their own secret scoring system and recommendations to get mail delivered, so to please them all, you have to: (i) set up an SPF record for the sending domain (a DNS record) - this involves stating which IP addresses send mail for the domain. If Hotmail (for example) see mail coming from an IP for org.org mail which is not included in the SPF record then it will be blocked. So for the purposes of mass mailing, it can be easier to send from a subdomain or new domain altogether, e.g. blocked.org.uk (ii) set up DKIM signatures for the domain on outbound mail (this is done in DNS and on the postfix server) (iii) set up reverse IP PTR records in DNS, such that the forward and reverse DNS lookups match. e.g. mail.org.org = 1.2.3.4, PTR(1.2.3.4) = mail.org.org. (this can be checked at mx tool box)

There might be other things, but that's the main ones. I can provide postfix config details

Sorry if teaching ORG to suck eggs!

[dantheta]

Not at all - It's been a while since I'd delved into the unpleasant realities of commercial email delivery.

I think since we're relaying through ORG's mail server, a lot of the IP characteristic and rate-related issues are best dealt with there. I think we can call this issue closed, and monitor for delivery problems downstream.