Group processors

[jiexinhuang]

Context

As an app using event_sourcery grows, it gets more and more event processors. How to manage and organize them is quite challenging, especially in a containerized environment. ESPRunner allows us to run multiple processors in one Ruby process, but it doesn't quite solve some of the problems we currently have:

• We cannot take down or bring up a subset of processors(if all processors are within 1 ESPRunner)
• Having different logging and alerting level for different processors(We want to wake on-caller up for critical errors, but some errors can wait until the next business day, it is a bit cumbersome to set the reporting level per error)
• Short deployment time(If we run each processor in its own container like what Identity does ATM, sequentially replacing all processor containers one by one is very slow)

Proposal

We want to group processors logically, so we can deploy each group individually, and take it down when needed.

Change

• Add processor_group class method to EventStreamProcessor
• A processor belongs to "default" group if no group name is set explicitly
• ESP registry can filter processors by type

[jiexinhuang]

Thanks @stevehodgkiss Is https://github.com/envato/event_sourcery/pull/179/commits/8c22ad7ef5e7ad5001a8a3f7a61bb3ae8dfa1bc3 be something you would expect? Maybe we can look into further extracting it out later.

[stevehodgkiss]

@jiexinhuang I was thinking the grouping logic would be completely independent of the ESP... so the ESP wouldn't know it's group, that would be defined using some other class elsewhere. Something like:

  run MyProcessor, as: 'my-processor'
  group :something do
    run SomethingProcessor, as: 'something'
  end

my_grouped_thing.start(:something)

The responsibility of the ESP shouldn't include the group it belongs to... that should be arranged elsewhere (and probably the runtime/processor name should live there too).

[twe4ked]

Can we get away without another DSL by doing something like this?

EventSourcery::EventProcessing::ESPRunner.new(
  event_processors: [FooProjector.new],
  event_source: EventSourcery::Postgres.config.event_source,
  group_name: :something,
).start!

Now that I've said that though I'm starting to think you could just have multiple instances of ESPRunner running if you want to group things.

Logging and alerting look to be handled at the EventSourcery.config level, you could set those before running the ESPRunner.

On a side note, in our app we've got an environment variable we can use to disable individual event processors. Changing an environment variable shuts down all our event processors and brings them back up (respecting the new settings).

[jiexinhuang]

I like the idea, @twe4ked Another thing I want to achieve is to avoid hardcoding a list of processors when starting ESPRunner, which is done by registry #by_group method. Now I feel this problem could have different solutions per app, I think I will implement it in our app instead. 😃