E-mail subscription to follow problems on specific sites

[hallvors]

Follow-up to #60, we should not only have per-domain RSS feeds but also a way to subscribe by E-mail to news about specific domains.

[hallvors]

Plan:

• [ ] Write the extra stuff required for the privacy policy
• [x] Add E-mail subscription database with fields [E-mail, domain, failures]
• [ ] Add code to the issue tagger to read subscription database when new issues come in and queue E-mail. (Depends on https://github.com/webcompat/web-bugs-issue-tagger/issues/10)
• [ ] Add some setup (separate app..) to send E-mail
• [ ] I need some help from @calexity and / or @magsout to make a nice UI for this feature.

The UI can either have two input fields "Your E-mail:" and "Site you want to keep an eye on" - or inside existing bug reports one field "Type your E-mail to be informed about new bugs from site ...:". I guess the latter is nicer UI-wise but it means people can only ever subscribe to sites we already have bugs for, so I want both types of UI.

(Subscription table might need some more data like date subscribed / date confirmed to let us sort-of prove we're not spamming people who didn't request a subscription.)

[karlcow]

• It requires to modify the Privacy policy
• Be careful too many things being processed at the time of the issue filling and we will slow down things and creates bottlenecks. Specifically with emails which requires a lot of time. Sending a job to a queue (in a different app) is fine. Handling the email right at the app level is a mistake.

[hallvors]

Thanks @karlcow , I've changed the todo-list accordingly.

This may be a question of architecture, and I don't have a full overview, but I'm wondering if sending E-mail from the webhooks handler really is a big problem. Will it slow down the entire site, or just slow down the response sent back to GitHub's bot? The latter is not a significant problem IMO :)

[karlcow]

Thanks @hallvors A couple of other things which came up during my jetlag night.

This requires also:

• User scenario for everything which can happen for the feature before writing code.
• Subscribe to what, where and when
• email notifications on why typye of events.
• Fully Unsubscribing
• Unsubscribing to one notification
• Knowing when an email address is not valid anymore or delayed because of servers snafu
• Admin UI of the email db
• Mass Email subscription (rate limit)
• Email verification through link to ack
• What information is provided in the email
• What actions are available in the email

And probably a couple more that my sleepy brain didn't get through.

[hallvors]

The main "user scenario" we want to provide for here is: "As a webmaster of example.com, Adam wants to be notified if a user reports an issue related to his site"

As any feature, it might also be used for other purposes (user with special interest monitors a given site, finance investors want to know what banks have most dissatisfied users) but the scenario we're developing to cater for is the one above.

Everything that can happen - from the user's point of view:

• Adam visits an issue related to his site, notices "Type your E-mail here to be notified about example.com reports" feature, types his E-mail
• Adam visits our "Site monitoring subscription" page, types in a hostname he wants to monitor ("example.com") and his E-mail
• Adam receives an E-mail confirmation message, requiring him to click a link with a secret nonce to confirm his subscription to notifications about example.com - site verifies the confirmation and says thanks
• Adam has been late at responding to the E-mail confirmation, nonce is no longer valid (I'd suggest one week validity). Site says the subscription attempt expired and links to the "Site monitoring subscription" page in case Adam wants to try again
• Adam receives an E-mail about a new issue with example.com but has left for a job at example.net instead. He clicks the easily accessible "unsubscribe" link, and gets a confirmation page asking "are you sure you want to stop notifications about example.com being sent to adam@example.com?" with an additional checkbox, unchecked by default, saying "Remove all notification subscriptions for adam@example.com"
• Clicking the "confirm" button, his subscription is deleted and the site confirms this.
• If "remove all" was checked, all rows related to the adam@example.com E-mail are removed from the database

So that's the user-visible tip of the feature. In particular, I don't think we need any sessions, user profiles, logins or "manage my subscriptions" dashboard at this stage. If we can read E-mail from user's GitHub profile data we can at a later point pull "subscriptions" into a user Dashboard, but it's better to develop this feature separately and avoid requiring users to have GitHub profiles and grant us access to them.

[hallvors]

[Edit: this comment was merged into the one above]

[hallvors]

Responding to the points that are not covered above:

• email notifications on what type of events.

Opening new issues for the specific monitored site. Nothing else - those who want to follow discussion and know when something is closed can get that from getting involved on Github.

• Knowing when an email address is not valid anymore or delayed because of servers snafu

Let's do a "three strikes and you're out" policy. We'll likely send few E-mails anyway, so the admin overhead should be low. This is implemented as a "failures" INT field in the database, per subscription, increasing the count from 0. The code that sends E-mail only uses E-mails with "failures" < 3.

• Admin UI of the email db

We should not need much here. Webcompat bounce processor person can have a secret admin page for pasting an E-mail address and increment its "failures" count - or resetting it to 0. Or just have SSH access and use a script that edits the db directly.

• Mass Email subscription (rate limit)

Hadn't thought of that but I guess we need to throttle / block subscription attempts if there's sabotage going on - like more than 10 / hour for a given IP..?

• What information is provided in the email

Hello, you have asked us to keep you informed of new issues reported for example.com. You may want to have a look at this incoming report: "example.com Desktop site instead of mobile site" https://webcompat.com/issues/0000

Please note that this issue was reported to us by a user and we have not verified that it is a genuine problem. If you can help us by adding comments or analysis to the issue it's much appreciated. You can do so either by using the "login" feature on webcompat.com and allowing us to use your GitHub profile, or by going directly to the GitHub issue tracker here: https://github.com/webcompat/web-bugs/issues/0000

Thank you for helping us monitoring example.com The webcompat team

PS: If you wish to stop subscribing to notifications about new reports for example.com, click here: https://webcompat.com/subscribe?delete=abcdefghijklmnopqrstuvwxyz

• What actions are available in the email

Link to issue and link to unsubscribe is all we need, I think.

[karlcow]

Wonderful @hallvors I wonder if @calexity has opinions about the proposed UX above.

[hallvors]

Here's some code to get us started: https://github.com/webcompat/webcompat.com/commit/47a16096e1349198daaf7f6d328e1460254aac44

This adds

• A mailsub.db SQLite file with a webcompat_subscribers table
• Table has these fields: row_id, domain, mail, secret, confirmed_timestamp, failures
• A /subscribe API URL which accepts mail= and domain= POST data and confirm=[secret] GET data
• A webcompat/mail component defining methods for sending E-mail. (If we need a separate "app" for this, the mail component can simply fill a DB table of E-mails to-be-sent or something like that)
• A new webcompat/static/mailtemplates folder with *.txt files corresponding to various E-mail messages we want to send

Criticism welcome (looking eastwards - in the general direction of Tokyo, @karlcow ;))

[hallvors]

(One thing I noticed now: defining /subscribe in api/endpoints.py means the URL ends up being webcompat.com/api/subscribe - right? OK for (Ajax) POST requests, but perhaps we don't want /api/ to show up in the confirmation- and unsubscribe URLs?)

[hallvors]

• [ ] Table also needs a requested_date timestamp field if we want to make the secrets expire
• [ ] /subscribe endpoint needs to handle delete=[secret] - show confirmation page and delete_confirm=[secret] - actually unsubscribe ppl.

[hallvors]

A thought: CAPTCHA? Pair browser logos and names correctly to continue subscribing? :-p

[hallvors]

Having thought about this some more, I don't want to add subscription forms to an issue - it will be too much clutter. I want just a link saying "subscribe to notifications for example.com" which takes you to a separate page with a form with a "domain" field pre-filled with "example.com" but editable.

I wonder if it would be a good idea to ask for the subscriber's name. I have a feeling that an E-mail including the user's full name may be less likely to end up in the spam filters.

As for needs-design; a draft of how the subscription form might look would be very useful.

[miketaylr]

For now, decided to WONTFIX in favor of #60