ESPRunner hook for responding to event processor failures

[orien]

We're encountering a problem when running our event processors via the ESPRunner. When one of the event processors running in a child process fails and terminates prematurely, the ESPRunner just ignores the problem. Eventually, our event processor lag monitor will raise the alert to the on-call developer, who in turn can manually restart the ESPRunner.

The process status list looks something like this:

> ps aux
USER     PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
app        1  0.0  1.6  90992 65204 ?        Ssl  03:40   0:02 ruby /app/script/processors
app        2  0.1  1.7  93624 68692 ?        Sl   03:40   0:04 MyApp::Queries::Projector1
app        3  0.0  0.0      0     0 ?        Z    03:40   0:03 [ruby] <defunct>
app        4  0.1  1.6  93640 67660 ?        Sl   03:40   0:05 MyApp::Queries::Projector3

We explored adding an extra option where the ESPRunner would shutdown in such a scenario in #215.

Change

This PR proposes a more extensible solution: provide a hook for the event processor failure. This'll allow teams to choose and implement an appropriate response as they see fit. Here're a few examples:

Report to Rollbar

  EventSourcery::EventProcessing::ESPRunner.new(
    event_processors: processors,
    event_source: source,
    after_subprocess_termination: proc do |processor:, runner:, exit_status:|
      if exit_status != 0
        Rollbar.error("Processor #{processor.processor_name} "\
                      "terminated with exit status #{exit_status}")
      end
    end
  ).start!

Shutdown the ESPRunner

  EventSourcery::EventProcessing::ESPRunner.new(
    event_processors: processors,
    event_source: source,
    after_subprocess_termination: proc do |processor:, runner:, exit_status:|
      runner.shutdown
    end
  ).start!

Restart the event processor

  EventSourcery::EventProcessing::ESPRunner.new(
    event_processors: processors,
    event_source: source,
    after_subprocess_termination: proc do |processor:, runner:, exit_status:|
      runner.start_processor(processor) unless runner.shutdown_requested?
    end
  ).start!