Proper notifications for event loop failure

[sattlerc]

Currently, the event loop has a hacky flag -e that allows you to specify a Google spreadsheet that will be edited with an exception message when the event loop fails. By subscribing to notifications on that spreadsheet, we can get failure notifications.

It would be better to solve this in a more standard way that doesn't rely on external services.

Now that we transition to using systemd to run the event loop as a (user-level) service, we can use systemd for this monitoring using the OnFailure feature. This should activate a unit that collects the exception from the end of the log and sends it by email to the local user. (In turn, the local user can configure forwarding addresses in .forward). A complication here: log messages may be multiline.

• [ ] Make the error stream output follow a well-defined format. This includes:
  • [ ] Update the event loop script to catch top-level exceptions and log them (with full trace) as an error message.
  • [ ] Use a log formatter that can make multiline log messages properly parsable (a search finds multiline-log-formatter).
• [ ] Write a script that extracts the trailing error message(s) in the service unit log (also useful for #25).
• [ ] Add a template for a systemd unit that collects the failure error and sends it to the local user.
• [ ] Configure OnFailure in systemd_unit_file.service.template to execute that unit.

[sattlerc]

@Niklas-Deworetzki Do you want to take a look at this? Peter can give you time.

[Niklas-Deworetzki]

Sure. The industry standard for monitoring would be to use Prometheus. We could start a prometheus client together with the event loop that reports different metrics (checked repositories, completed/rejected outgoing connections, number of tags created by students). But that might be a bit overkill in this case.

I think the error reporting is better done as part of the script itself. If we catch top-level error messages in the event loop, we can also try to alert an administrator for them. There is probably more we can do in Python than by using some black magic and systemd.

[sattlerc]

I think the error reporting is better done as part of the script itself. If we catch top-level error messages in the event loop, we can also try to alert an administrator for them. There is probably more we can do in Python than by using some black magic and systemd.

How about this?

• [ ] Implement a mechanism in run_event_loop.py for catching top-level exceptions and emailing them to some specified email address(es).
• [ ] Add flags to run_event_loop.py that expose this functionality (eventually to replace '-e'). By default, it could send them to the local user.
• [ ] Test this on the lab server.

[sattlerc]

Who is working on this?