search

envato / event_sourcery

A library for building event sourced applications in Ruby
MIT License
84 stars 10 forks source link

Gracefully shutdown event stream processors #97

Closed orien closed 7 years ago

orien commented 7 years ago

EventSourcery has an issue (#82) where on reactor shutdown (upon receiving a TERM signal) it will quit immediately, potentially in the middle of processing an event. This can result in an action being taken (email sent, event emitted etc) and the tracker not being updated. When the system starts again the event in question will be processed a second time and the action duplicated.

This proposal alleviates the problem by gracefully handling the TERM signal. The process records that shutdown is desired and waits till a safe time, in between processing events, to quit. If the the process takes too long to quit gracefully it is sent a KILL signal. Upon which it quits immediately similar to the current behaviour.

The graceful shutdown is implemented via a throw command, corresponding to an already existing catch block deep in the event polling code. I'm not a fan unstructured approaches such as this. However it is the simplest approach at the moment, avoiding a big refactor of the event polling code.

mjward commented 7 years ago

Someone has been busy ;) nice work!

twe4ked commented 7 years ago

Looking good 💯

orien commented 7 years ago

@stevehodgkiss Sure could use ESPProcess on it's own. You wouldn't get the timeout/kill logic. But there are many other things you can use configure to do this, Foreman or systemd for instance.

orien commented 7 years ago

Basic tests look good on Payables. This app takes 5-6 seconds to shutdown.

twe4ked commented 7 years ago

🎉