review how to detect failed score updates

[matthewcornell]

...and about notifications of problems in general. Examples:

• a model/score combination failed to finish
• a forecast failed to upload

Possible solutions:

• checking by hand:
  • Job.status (uploads, queries, etc.) - is done by libraries' busy_poll() functions
  • failed RQ jobs - check RQ admin failed jobs queue
  • logs :-/

At a higher level, we need to do a systematic review of all the kinds of production errors we regularly see, and make a plan for mitigating them.

[matthewcornell]

Here's a simple solution: Use Papertrail's alerts feature:

1. Save a search in Events, e.g., JobTimeoutException
2. Setup an alert, e.g., with Slack

We should probably make log messages more consistent - currently the messages vary depending on the error, such as here. Maybe something like <function_name>(): error: <error class> ....

• "job_cloud_file(): Job timeout"
• "job_cloud_file(): AWS error"
• "job_cloud_file(): Error"
• "_query_worker(): Job timeout"
• "_query_worker(): error running query"
• "_update_project_row_count_cache_worker(): Job timeout"
• "_update_project_row_count_cache_worker(): error"
• "_update_model_scores_worker(): Job timeout"
• "_update_model_scores_worker(): error"
• "_upload_truth_worker(): Job timeout"
• "_upload_truth_worker(): exception"
• "_upload_forecast_worker(): Job timeout"
• "_upload_forecast_worker(): exception"
• "_delete_forecast_worker(): Job timeout"
• "_delete_forecast_worker(): error"

Links: Papertrail integration with Slack:

• https://papertrailapp.com/alerts
• https://documentation.solarwinds.com/en/Success_Center/papertrail/Content/kb/how-it-works/alerts.htm?cshid=pt-how-it-works-alerts
• https://documentation.solarwinds.com/en/Success_Center/papertrail/Content/kb/integrations/slack.htm?cshid=pt-integrations-slack
• https://reichlab.slack.com/apps/A0F81FMT8-papertrail