Improve the FailedRunsNotificationCronJob to be more resilient

[evenicoulddoit]

In it's existing setup, the FailedRunsNotificationCronJob simply sends an email when the number of failed runs for a certain job reaches a pre-determined level (which defaults to 10). After this point (i.e. at fail 11, 12...) the user will not receive any further emails.

Furthermore, the job will also continue to send emails to the users every time the runcrons task is called, provided the number of failed runs remains at the level which triggers the email. Therefore, if you're using crontab e.g. to run the runcrons management command every 10 minutes, the user will be emailed about the failure every 10 minutes.

101 aims to fix both problems. By adding a failure_reported flag to the CronJobLog model, Once the number of failures exceeds the user determined limit, they are marked as having been notified.

In addition to the above, a few changes are made to:

• Provide more settings to modify the email recipients
• Allow the FailedRunsNotificationCronJob to be extended in order to modify how the notification takes place (e.g. using Slack, text etc..)

Tests are provided and documentation has been added

[tab-cmd]

@evenicoulddoit thanks, will review shortly 👍