System for emailing job failures

[rhettg]

Let's create a system for sending out emails based on the status of running jobs. This will be a combination of infrastructure for sending emails based on some events, and the configuration options for setting them up.

User should be able to configure, on a per-job basis:

• Email address to send report to
• When to send an email (Job failure, Job success, Job stderr)

Email should contain (when applicable):

• Job ID
• Execution Node
• Job/Action status (success, failure)

• Job/Action runtimes (start, stop, duration)

  For bonus points, we should provide support for the following:

• Sending emails when a job's duration has exceeded some configurable threshold
• Sending email if a job is ever queued

These could be done as a separate feature, but keep them in mind.

[dodizzle]

Would be great if there could be different alert levels based on exit code just like in nagios. exit 0 = OK exit 1 = warming exit >1 = failure