[Epic] Sending Emails when flow is broken

[zelima]

As a publisher I want to get notified with something is broken to the automated dataset, so that I can take a look an fix it

Acceptance Criteria

• [x] We are getting the email with notification that pipeline has been broken for core

Tasks

• [x] Do analysis if we can integrate open source projects for getting alerts
  • [x] analysis for status.io
  • [x] analysis for statuspage
  • [x] analysis for Cachet
• [x] Analysis for using dpp hooks
  • [x] Understand how dpp hooks work and how can we use
• [x] Do analysis if we can integrate IFTTP
• [x] Create account on https://status.io (if free) https://github.com/datahq/pm/issues/286
  • [x] Play around with their API
• [x] Create an account on https://statuspage.io (free trail) https://github.com/datahq/pm/issues/287
  • [x] Play around with their API
• [x] Check out pingdom.com and https://apex.sh/ping/
• [x] Add a method on_incident that send Email to a user with a message https://github.com/datahq/datahub-emails/issues/1
  • [x] may also needed subscribe to add user emails
• [ ] Decide which datasets are we going to monitor (certified?)
  • [x] Create components manually for now (core and sports-data)
• [x] refactor specstore to https://github.com/datahq/specstore/issues/50
  • [x] use on_incident on failure to send Email
  • [ ] use subscribe to subscribe for now do it manually (?)
• [x] Subscribe to the cheepest plan for now https://www.statuspage.io/pricing
• [x] update specstore envs with statuspage api-key https://github.com/datahq/deploy/issues/160
• [x] Tidy up statuspage dashboard

Analysis

Question: What do we really need here? What is the main user story?

• I published my dataset once and it's getting updated every X days ->
• After X days the processors will re-run and I don't know whether it runs fine or not (I'm not even sure how to check tbh)
• So I want to know if this or that run went fine and get alerted if not
• Not this: I published once, and every once in a while check if it's still loading fine when visiting datahub.
  • Eg I published a dataset that should be updated annually. I want to get alerted if something went wrong on an annual run, and not Eg check weekly if dataset page is loaded fine and alert me if not

Getting the Emails

Q: How should we add users to the list/segments? A: Both status.io and statuspage.io have a method to add Email to the subscriber's list to the incidents

Q: Who exactly should get the emails? A: I assume this service is not for everybody, so we should have some kind of flag in the database indicating so. Or alternatively we may create list of subscribed users manually and assign to the incidents

Q: Will this be automated? or manual for now? A: Really depends on who we want to send emails. If we want to define them manually than we should create them manually. But both APIs allow us to create subscribers list via API. A: Though we will need to update DB structure to flag that user is subscribed for failure. Or for now we can use certified flag and if true send notification user as well.

What info should users get?

Q: What exactly does something failed mean? A:

• Processors failed
• Validation errors
• Limit exceeded
• TimeOut(?) - we do not monitor this yet (though useful as some of the datasets keep hanging and keep loading forever)
• ... (?)

Technical implementation

WIP

Think we can add new method to datahub-email and use it in flowmanager (specstore).

...
if flow_status = STATE_FAILED or len errors:
  emails.send_incident(user.email)
...

Services

https://apex.sh/ping/

BASIC | PLUS | PRO
-- | -- | --
MONITORSNumber of URLs which can be monitored. | 15 | 50 | 100
TEAM MEMBERSNumber of team members who can view and manage monitors, alerts, and status pages. | 5 | 10 | 20
ALERT RULESNumber of alert rules for triggering downtime or performance notifications. | 2 | 10 | 30
REGIONSNumber of regions a single monitor will perform requests from. | 15 | 15 | 15
STATUS PAGESNumber of status pages you may create. Typically you'd have one per domain, such as status.apex.sh. | ∞ | ∞ | ∞
  | 10 | 30 | 60

This one seems to have all in one and is relativly cheept. No public API available though, we should build everything manually. Won't be abale to create incident on failure

Pingdom (https://www.pingdom.com/)

Starter | Standard | Advanced | Professional
$11.95/mo | $36.00/mo | $72.00/mo | $199/mo
-- | -- | -- | --

This one is really awesome - is monitoring the whole aplication in live, checking speed, states, performence etc... But not quite what we need for this task. It does not allow us to create custom incidents and alert users about it.

https://status.io/

• Does not have free plan (not even free trail): https://status.io/pricing

  Basic Standard    Plus
  $79/mo    $149/mo $349/mo

• Does have python API https://github.com/statusio/statusio-python
• delivery Methods: Email, SMS, Webhook, RSS, iCalendar, IRC, HipChat, Slack, Twitter
• Custom emails not included in basic plan
• Custom html starting from Plus plan

All plans include

• One Public Status Page
• Custom Domain
• Secure SSL / TLS Subdomain
• SSO / Two-factor Authentication
• Customizable Design
• Incident and Maintenance Templates
• Granular Subscriptions
• Custom Email Notification Templates
• Developer API
• Public Status API
• History Search and Filtering
• Super Friendly Support
• Learn more about our feature set

Authentication pretty simple, only provided API _DI and API_KEYs are required. Once authenticated there are several useful methods available:

import statusio
api = statusio.Api(api_id='api_id', api_key='api_key')

api.IncidentCreate(statuspage_id, notify_email=0, ...)
api.SubscriberList(statuspage_id)
api.SubscriberAdd(statuspage_id, method, address, silent=1, granular='')
api.SubscriberUpdate(statuspage_id, subscriber_id, address, granular='')
api.SubscriberRemove(statuspage_id, subscriber_id)
...

I assume statuspage_id is assigned to each status page. Also meaning multiple status page may be created Eg: for each dataset (Not sure thoug)

Think we can start experimenting with it

https://www.statuspage.io/

I like this one most

• Does not have free plan (has free trail): https://www.statuspage.io/pricing

  Hobby  Startup  Business  Enterprise
  $29  $99  $399  $1,499

Hobby

• 250 Subscribers

• 2 Team Members

• 2 Metrics

• Email Notifications

• Basic Customization

• Status & Authenticated API

• Has RestAPI https://doers.statuspage.io/api/v1/component_groups/

• delivery methods: Email, SMS, or Webhook

Simple, Authentication with API key in request header.

Useful API endpoints

POST https://api.statuspage.io/v1/pages/{page_id}/incidents/{incident_id}/subscribers
GET https://api.statuspage.io/v1/pages/{page_id}/incidents/{incident_id}/subscribers

POST https://api.statuspage.io/v1/pages/{page_id}/incidents

This to (status page and status io) seem quite similar in their functionality and API. So we can choose any of them.

• This one has free trial (prob status.io also has but need registration upon)
• but the other one python wrapper

http://cachethq.io/

Not sure how exactly we can use this and don't really like it tbh:

• Can't find pricing page here at all
• It seems a bit complicated and messy to me.
• JSON API
• Subscriber notifications via email
• Not sure but think needs PHP to be installed

IFTTP

Don't really understand how this might work with our services. Seems to me if I sit down and really dig down to their docs it will take much more time than any of listed above so skipping at this point

[zelima]

FIXED