Receive email notifications when a player receives game messages or games progress

[tsani]

Right now, it's necessary to check on the game very often to keep up with messages. In a fast paced game, that's certainly fine, but in a slower paced game, email notifications would be a great way to avoid having to poll the game page for updates every now and then.

[TimothyJones]

This is an often requested feature. I'm going to extend this a bit to configurable email notifications for:

• In game messages
• Phase progressing and adjudication results
• Votes
• PMs
• When a game you are in starts

The default should probably be "off" for at least all existing members, since many members have also said that they don't want this feature.

[TimothyJones]

Care needs to be taken to avoid our emails being detected as spam.

[thorfi]

Yo. :-) So, as far as I can tell, webdiplomacy.net's email is a "Google Apps" domain. So the relevant bits are:

limits for individual users: https://support.google.com/a/answer/166852

limits for "relaying" through google apps (i.e., using google apps as a "smarthost" in mail sender terms):

https://support.google.com/a/answer/2956491

I'm not sure which of these the web dip server is using. It could be either. It would probably be better to configure the google apps domain to do the latter rather than the former, since the sender limits are much much higher than that for an individual account. It should be possible to configure the web dip server to handle sending using the latter rather than the former without running a local mail server, if that's not already what it's doing.

As far as being flagged as spam goes - it's actually not that likely, especially if you're sending out from google, rather than sending out from your own mail server.

One other thing is that it is probably worth adding an "SPF record" to the DNS for web diplomacy.net:

TXT "spf2.0/pra"
TXT "v=spf include:_spf.google.com ~all"

How to add TXT records depends on who is doing your DNS hosting, I'm afraid.

(And yeah, the second line is the real one, the first one is to deal with Microsoft Exchange brokenness - don't ask, it's a tentacled horror you don't want to know about.)

[TimothyJones]

Great info, thanks @thorfi. @kestasjk , what are your thoughts on this?

[thorfi]

I would also suggest that a monthly/quarterly/yearly "Hi, you're on web diplomacy, here's all the games you're in" notification email might be worth doing and defaulting that to actually on, rather than off?

You may also need to consider if and how to handle email bounces from notifications.

[TimothyJones]

I'm not sure about a yearly email reminder - what purpose would it serve that the regular updates (or checking the site) doesn't?

I think it makes sense to default email notifications to off for existing users (since they haven't agreed to receive the messages), but on for new members (since a lot of new members expect email notifications anyway, and the signup page is the same as the user settings page - so they can easily change the setting there if they want). This should be pretty easy to do with the new options mechanism.

[thorfi]

Some sort of long-periodic notification could: a) possibly re-activate idle users b) serve as a check for registered email address validity (especially in combination with some usefully automated method of handling email bounces). c) maybe could include "new features since last notification" announcements

Possible downsides: a) As you say, sending email to people who haven't necessarily opted in might be problematic. b) bounce handling is... um, a complicated feature to implement (but you might need it anyway - if you send out a lot of email that bounces, I expect gmail might not like that very much eventually).

[TimothyJones]

I don't anticipate too many invalid emails; signup requires an email step, so the emails were valid at some point.

You're right that a principled way of handling bounces is a good idea.

[thorfi]

The other option if you want to avoid possible issues with bounce handling is to actually deploy and run a mail server instead of using google apps for outgoing mail. That also gives the option of doing customised email addresses to simplify automated bounce handling and such things. But that all gets... potentially complicated, and running your own mail server is sort of not that trivial.

[thorfi]

I think you may be surprised at how rapidly email addresses go invalid. :-)

[Sleepcap]

Maybe a reactivation-link like in the registration-mail that you need to reconfirm your email every once in a while would help to ensure valid email-addresses too. You do not need to block the complete account without a confirmation-link, but maybe block joining or creating new games (and/or forum posts) for unvalidated accounts.

[Sleepcap]

Or maybe make a time stamp for each account and every 6 or 12 month the user need to verify his email-address again if he want to continue to play. That way you do not need to automate this for all players, but you can add the code in the game join/create-code. Also the players know there is a new email coming, so they will respond very fast.

[thorfi]

Ah yes. So the general procedure for most sites that do email like this is:

1. Verify email address (or addresses) using activation link emails
2. Do specific user bounce detection (by sending emails to users using a user-unique ENVELOPE FROM (NB: This is not the From: header inside the email, this is the user specified in "MAIL FROM:<foo@example.invalid>" during SMTP delivery) then inspecting and handling bounces that arrive).
3. If a user gets a bounce, flag that address as invalid and notify the user via a header bar that they may wish to re-validate their address when they log into the website

As far as 2. goes, for web diplomacy, gmail supports address+extrabit@, so if you set the envelope from address as "outgoing+useruniqueslug@webdiplomacy.net" when you send an email, bounces should end up in the mailbox of outgoing@webdiplomacy.net, and a script could log in to that via IMAPS, grab any emails and inspect them for bounce details and then do the right thing, possibly including forwarding any not-understood email to someone else. I would not make useruniqueslug the user ID - but maybe the user ID plus hash of the userID or UserName, or even an actually randomly generated piece of text stored in the user table, just so they're not sequential.

Alternately, perhaps consider migrating all outgoing mail to Mandrill (the MailChimp service back end):

https://mandrill.com/features/ https://mandrill.com/pricing/ ( first 12k emails per month are free, suspect webDip is below that threshold )

[TimothyJones]

We can work out exactly how many emails per month we need by looking at the contents of the notifications table. I'll have a look tonight - could be pretty interesting.

[thorfi]

1. Out of curiosity, did ^this db query get done and what were the results?
2. Of possibly related interest, there are two potential alternate mechanisms for notification delivery:

2.a. "Web Notifications" API - http://www.w3.org/TR/notifications/ requires an open web page, and running javascript. I believe most desktop browsers implement the window.Notification object now. I am not sure this is well supported on mobile browsers, however.

2.b. Safari Push Notifications (OS X 10.9 Mavericks or later only) - https://developer.apple.com/library/mac/documentation/NetworkingInternet/Conceptual/NotificationProgrammingGuideForWebsites/Introduction/Introduction.html That page links to descriptions for stuff in 2.a. as well. Safari Push Notifications do not require an open web page, but obviously they are only available for OS X users, rather than all users.

I'm not sure if either of these is any use, but they exist so I thought I'd mention them. :-)

[TheDevilOnLine]

I'm willing to write some code that allows this to function through push notifications, as I'd love to have notifications. Alternative would be a chrome/firefox extension.

Let me know if this is something you would accept.

[jmo1121109]

Absolutely @TheDevilOnLine

[diptobi1]

Not aware of this conversation I have recently started looking into push notifications for webdip, too. Though as I am not a professional and just working out how to do this, I would be happy to pass this task to a more experienced person. Otherwise I might be able to come up with an implementation soon, as well.

[jmo1121109]

If either of you need any help with this or if you end up working together on it and need support please let me know. I haven't worked with push notifications before, but am happy to assist with any architecture changes needed to make this a reality since it would be an amazing improvement for the site.

[nampas]

I would also be very interested in email notifications on game progress. I'm happy to contribute, if the maintainers are open to it? I have experience with email and push notifications, and a reasonable amount of experience with php :)

[jmo1121109]

@nampas we would absolutely be open to it. I don't think our current email service has the capacity to deal with the volume of emails this would include though so if you know of decent services that would be really helpful too.

[nampas]

Ok, let me look around the code a bit and situate myself. I'll then outline an approach in this ticket, potentially incorporating another email service.

[borisjoffe]

As a stopgap, why don't we use RSS feeds?

It doesn't require any email integration / fees.

And we could backport the implementation from the vDiplomacy fork (https://github.com/Sleepcap/vDiplomacy).

RSS is a nice, open, standards-compliant solution and can eventually be improved with email, push notification, and sms functionality. There are also existing tools that users can set up to do RSS-to-email, email-to-SMS, and email-to-push-notification